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FOREWORD 


The Software Engineering Laboratory (SEL) is an organization 
sponsored by the National Aeronautics and Space Administra- 
tion Goddard Space Plight Center (NASA/GSFC) and created for 
the purpose of investigating the of fcctivcners of software 
engineering technologies v/hen applied to the development of 
applications softv;are. The SEL was created in 1977 and has 
three primary organizational members: 

NASA/GSFC (Systems Development and Analysis Branch) 

The University of Maryland (Computer Sciences Department) 

Computer Sciences Corporation (Flight Systems Operation) 

The goals of the SEL are (1) to understand the software de- 
velopment process in the GSFC environment; (2) to measure 
the effect of various methodologies, tools, and models on 
this process; and (3) to identify and then to apply success- 
ful development practices. The activities, findings, and 
recommendations of the SEL are recorded in the Software En- 
gineering Laboratory Series, a continuing series of reports 
that includes this document. A version of this document was 
also issued as Computer Sciences Corporation document 
CSC/TM-83/6019. 
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ABSTRACT 


This document presents a set of guidelines for an organized, 
disciplined approach to softv^aro development, based on data 
collected and studied by the Softv/are Engineering Laboratory 
(SEL) since 1977 for 46 flight dynamics software development 
projects. It describes methods and practices for each phase 
of a softvjare development life cycle that starts with re- 
quirements analysis and ends with acceptance testing; main- 
tenance and operation is not addressed. For each defined 
life cycle phase, this document presents guidelines for the 
development process and its management, and the products 
produced and their reviews. This document is a major revi- 
sion of SEL-Bl-105. 
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The Software Engineering Laboratory (SEL) was established in 
1977 by the National ^Aeronautics and Space Administration 
Goddard Space Flight Center (MASA/GSFC) to investigate the 
effectiveness of softwcire engineering techniques applied in 
developing ground support flight dynamics systems. The 
investigation's goals are (1) to understand the software de- 
velopment process in a particular environment? (2) to measure 
the effects of various development techniques, models, and 
tools on thin development process; and (3) to identify and 
apply improved methodologies in the GSFC environment. The 
results of SEL research should enable GSFC to produce better, 
less costly software. 

This document presents a set of softv/are development guide- 
lines for a disciplined approach to software development, 
with special emphasis on management considerations. These 
recommendations are based on data collected and studied by 
the SEL since 1977 for 46 flight dynamics software develop- 
ment projects. Most of the softv/are development projects 
studied by the SEL v/ere performed by an independent con- 
tractor. For some projects, however, the software develop- 
ment team consisted of both GSFC and contractor personnel. 
Developing a flight dynamics software system involves di- 
verse skills and many staff-years of effort in specifying, 
designing, implementing, integrating, and testing complex 
computer programs. This document describes the software 
development life cycle from the technical manager's per- 
spective and provides useful techniques for managing soft- 
ware development. 

This document is neither a manual on applying the technol- 
ogies described here nor a tutorial on monitoring a Govern- 
ment contract. Instead, it describes the methodologies and 
tools that the SEL recommends for use in each life cycle 
phase to produce reliable, cost-effective software. 
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Introduction 


This document is primarily for technical managers of soft- 
ware development efforts. However, it is also intended for 
higher level managers v;ho are concerned v/ith schedules and 
budgets and for senior technical personnel (such as de- 
signers, analysts, and programmers) who are responsible for 
implementing the recommended procedures. The recommended 
softv;are development guidelines are appropriate for both 
GSFC and contractor personnel. Although the recommended 
guidelines have been formulated based on the development of 
flight dynamics systems, there is no reason to believe that 
the recommended guidelines are not applicable to any 
software development project. 

The SEL continually monitors and studies flight dynamics sup- 
port software, including software developed by both GSFC 
employees and contractor personnel. It anticipates that 
data will continue to be collected and analyzed in the fu- 
ture. The recommendations in this document will be refined 
and enhanced as more knowledge is obtained about the produc- 
tion of better, less costly softv/are and as more progress is 
made toward achieving the goals of the SEL. 

1.1 DOCUMENT OVERVIEW 

Section 1 describes the document's purpose and its intended 
audience and establishes the general background for the re- 
mainder of the document. Sections 1.2 and 1.3 briefly de- 
scribe the SEL and the flight dynamics softv/are development 
environment. 

Section 2 describes the typical software development life 
cycle in the flight dynamics area, identifying the major 
phases and their characteristics. 

Section 3 describes in detail the recommended guidelines for 
software development. It discusses the major activities, 
end products, methodologies, tools, and measures applicable 
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to each life cycle phase. In addition, it recommends man- 
agement activities for each life cycle phase. 

Section 4 discusses management and control of the develop- 
ment process, and Section 5 summarizes key aspects of suc- 
cessful and unsuccessful projects. Appendix A discusses 
formal reviews. Appendix B presents the content and format 
of documents, and Appendix C provides a brief example of 
some steps in organizing a project. Appendix D summarizes 
the key information of the document. 

A glossary, references, and a bibliography of SEL literature 
are also included. 

1.2 SOFTOARE ENGINEERING LABORATORY 

The SEL monitors and studies all software developed by the 
Systems Development Section at NASA/GSFC, which is respon- 
sible for producing flight dynamics support software for 
GSFC-supported missions. To date, 46 projects developed by 
both GSFC and contractor employees have been studied. They 
range in size from 1,500 lines of source code to more than 
110,000 lines. Much of the data is collected on a series of 
forms completed by project personnel throughout the develop- 
ment effort. Data is also collected through computer ac- 
counting monitoring, personal interviews, automated tools, 
and summary management reviews. From investigating projects 
totaling more than 1.5 million lines of source code, the SEL 
has gained insight into the software development process and 
has begun to identify trends in the effects of applying var- 
ious techniques to the software development projects. 

The SEL approach to software engineering research is based 
on the high-level softv/are development model shown in Fig- 
ure 1-1. The four components of this model are a problem 
statement, an environment, a process or activity, and a 
product (software) . The development process is divided into 
seven sequential phases of activity. These phases are de- 
scribed in more detail in Section 2. 
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SEL reso-irch first attempts to understand the software de- 
velopment process currently In operation and its environment. 
This understanding provides a baseline for measuring the 
effects of attempted improvements. Next, the SRL tries to 
improve that process and environment to produce high-quality 
software with fewer errors at a lower cost. To achieve 
these goals, the SKL identifies the development techniques 
available, evaluates these techniques to determine the most 
effective ones, adapts the "best" techniques for optimal 
performance, and applies the customized techniques to the 
software development process. The recommendations in this 
document are based on the application of this four-step 
procedure to a large set of software development technol- 
og ies. 

The technologies studied by the SEL may be classified into 
four maior areas of concern; methodologies, tools, models, 
and measures. Methodologies are systematic applications of 
proscribed principles to the development process. Tools are 
software aids used during the development process to make it 
easier for development team members to do their work. Models 
explain and/or predict some aspect of the development proc- 
ess and are usually formulated as mathematical equations 
relating two or more quantitative factors. Measures define, 
explain, and predict software development qualities; they 
may be objective or subjective. 

Section 3 identifies the technologies in these four areas 
that the SEL recommends for application to the software de- 
velopment process during the development life cycle phases. 

Reference 1 contains additional information on the activ- 
ities of the SEIi and further details on the results of 3F.L 
research . 
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FLIGHT DYNAMICS ENVIRONMENT 
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The empirical basis for this document is the studies of 
flight dynamics software conducted by the SEL since 1977. 
Flight dynamics software includes applications to support 
attitude determination, attitude control, maneuver planning, 
orbit adjustment, and general mission analysis. The atti- 
tude systems, in particular, form a large and homogeneous 
group of software that has been studied extensively. 

The attitude determination and control systems are designed 
similarly for each mission using a standard executive sup- 
port package, the Graphic Executive Support System (GESS) , 
as the controlling system. All these systems are designed 
to run in batch and/or interactive graphic mode. Depending 
on mission characteristics (for example, the type of data 
available and the accuracy required) , these systems may 
range from 30,000 to approximately 120,000 lines of code. 

The percentage of reused code ranges from 10 percent to 
nearly 70 percent, with the average system reusing about 
30 percent. 

The applications developed in the flight dynamics area are 
mostly scientific and mathematical in nature, with moderate 
reliability requirements. Severe development time con- 
straints are imposed by the spacecraft launch date; the 
software mur** be completed (through acceptance testing) 

90 days befc the scheduled launch, and the requirements 
and functional specifications are normally made available 
less than 2 years before the scheduled launch. 

Most flight dynamics software projects use a group of IBM 
S/360 computers for development. All resources on these 
machines are extremely limited, and the hardware is very 
unreliable.^ Because the machines are shared among the 


^The mean time between failures for the primary development 
machine is approximately 6 to 8 hours of operation (see Fig- 
ure 3-4 of Reference 1) . 
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analysis, software development, and operations areas, soft- 
ware development schedules are affected when simulations, 
launches, and maneuvers occur. In addition to the IBM 
S/360S, a DEC PDP-11/70 and a DEC VAX-11/780 are occasion- 
ally used to develop utilities and support systems for the 
flight dynamics area. 

Additional information about the flight dynamics software 
development environment may be found in Reference 1, Sec- 
tion 3.1 of that document presents the results of profile 
analysis of several large softv/are development projects 
studied by the SEL. These statistical profiles characterize 
the software development process, environment, and products 
for these flight dynamics projects. 
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The flight dynamics . software development process is divided 
into the following seven sequential phases, collectively 
referred to as the software development life cycle: 

1. Requirements analysis 

2. Preliminary design 

3. Detailed design 

4. Implementation (code and unit testing) 

5. System testing (system integration and testing) 

6. Acceptance testing 

7. Maintenance and operation 

This division is shown in Figure 1-1 (page 1-5), which il- 
lustrates the SEL software development model. 

For the purpose of this document, the software development 
life cvcle is divided into calendar phases rather than ac- 
tivity chases; that is, the seven life cycle phases sub- 
divide the software development effort into seven sequential 
periods of time that do not overlap. Each calendar phase of 
the software development life cycle is characterized by 
specific activities and the products produced by those ac- 
tivities. 

Activities that are characteristic of one calendar phase, 
however, may be performed in other phases. For instance, 
analvzing the requirements, which makes up most of the 
effort during the requirements analysis phase, continues at 
a lower level throughout the software development life 
cycle. Once a development activity is started, it continues 
throughout the life cvcle; however, the level of effort for 
early life cycle activities continually decreases. SET, data 
shows that the estimated size of large flight dynamics 
systems grows between 15 and 40 percent after implementation 
starts (that is, after completion of the detailed design) 
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because of uncertainty in the estimation process , new 
requirements, and requirements changes. This grov;th causes 
requirements analysis and design activities to continue 
during subsequent life cycle phases. Figure 2-1 illustrates 
the activities performed during each (calendar) life cycle 
phase as a percentage of the total staff effort. 

The following subsections define the seven software develop- 
ment lifG cycle phases, and Section 3 describes them in more 
detail. 

2.1 REQUIREMENTS ANALYSIS 

Before the requirements analysis phase begins, a team other 
than the development team defines the requirements and pro- 
duces a functional specifications and requirements document. 
The requirements analysis phase begins when the requirements 
definition team completes the draft of the functional speci- 
fications and requirements document. It is the first phase 
of the software development life cycle, and in it, the func- 
tional specifications are translated from mission terms into 
a softvjare-supportable form. 

In this phase, the development team analyzes the functional 
specifications and requirements document from a software 
system viewpoint and recasts the requirements in terms suit- 
able for software design. The development team assesses the 
completeness and feasibility of the requirements, identifies 
missing or to-be-determined (TBD) requirements, specifies 
all external interfaces, and makes the initial determination 
and allocation of resources. Close interaction with the 
requirements definition team is necessary for the develop- 
ment team to clarify and amplify the requirements. The de- 
velopment team gives the results of the requirements analysis 


^See Table C-1 on page C-4 of Appendix C and Figure C-1 on 
page C-8 for information on uncertainty limits. 
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to the requirements definition team for incorporation into 
the final version of the functional specifications and re- 
quirements document. They also prepare a summary report of 
the results as a basis for preliminary design. Vlhon the 
final version of the functional specifications and require- 
ments document has been completed, a system requirements 
review (SRR) is held to evaluate the completeness of the 
requirements. 

2.2 PRELIMINARY DESIGN 

During the preliminary design phase, the development team 
defines the software system architecture based on the re- 
quirements given in the functional specifications and re- 
quirements document. The team translates this architecture 
into software requirements in the requirements analysis sum- 
mary report. During this phase, the development team spec- 
ifies major functional subsystems, input/output interfaces, 
processing modes, and implementation strategy. The require- 
ments evaluated during the requirements analysis phase are 
translated into functional capabilities and are organized 
into major subsystems. All internal and external interfaces 
are completely defined to the subsystem level, and the de- 
sign is refined to two levels below the subsystem drivers. 
Figure 2-2 illustrates the hierarchical levels of a software 
system baseline diagram (treechart). The development team 
documents the functional design of the system in the prelim- 
inary design report. The preliminary design phase culmi- 
nates in the preliminary design review (PDR) , where the 
development team formally presents the functional design for 
review. The preliminary design is considered complete when 
responses to the PDR comments and criticisms have been in- 
corporated in the functional design. 
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DETAILED DESIGN 


During the detailed design phase» the development team ex- 
tends the system architecture defined in preliminary design 
to the subroutine level. By successive refinement tech- 
niques, they elaborate the preliminary design to produce 
"code-to" specifications for the system. 

All formalisms for the system design specifications are pro- 
duced, including functional and procedural descriptions of 
the system; data flow descriptions; complete descriptions of 
all user input, system output (for example, screen, printer, 
and plotter), and input/output files; operational procedures; 
functional and procedural descriptions of each module; com- 
plete descriptions of all internal interfaces between mod- 
ules; and build/release capabilities. The team documents 
these design specifications in the detailed design document, 
which forms the basis for implementation. The detailed 
design phase culminates in the critical design review (CDR) , 
where the development team formally presents the "code-to" 
specifications for review. The detailed design is consid- 
ered complete when the responses to the CDR comments and 
criticisms have been incorporated j n the detailed design 
document. 

2.4 IMPLEMENTATION (CODE AND UNIT TESTING) 

In the implementation (code and unit testing) phase, the 
developers code new modules from the design specifications 
or revise old code to meet new requirements, integrate each 
module into the growing system, and perform unit and inte- 
gration testing to ensure that the newly added capabilities 
function correctly. 

In a typical project, the developers build several subsys- 
tems simultaneously from individual components. The team 
repeatedly tests each subsystem as new components are coded 
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and integrated into the evolving software. At intervals, 
they combine subsystem capabilities into a complete working 
system for testing end-to-end processing capabilities. The 
sequence in which functions are coded and integrated into 
executable subsystems and the process of combining these 
builds into systems are defined in the implementation plan 
produced during detailed design by the development managers. 

The team also produces a system test plan and drafts of the 
user's guide and system description documents during this 
phase in preparation for the system integration and testing 
phase that follows. Implementation is considered complete 
when all code for the system is produced, tested, and inte- 
grated into the system. 

An independent acceptance test team prepares the acceptance 
test plan based on the information in the functional speci- 
fications and requirements document. The acceptance test 
team usually consists of analysts who will use the system, 
and it frequently includes members of the organization that 
specified the requirements. 

2. 5 SYSTEM TESTING (SYSTEM INTEGRATION AND TESTING) 

During thr system testing (system integration and testing) 
phase, the development team validates the completely in- 
tegrated system produced by the implementation phase. This 
means that functional testing of end-to-end system capa- 
bilities is performed according to thfe system test plan 
developed during the preceding phase. The system test plan 
is based on requirements set forth in the functional speci- 
fications and requirements document. Successfully complet- 
ing the tests specified in the test plan demonstrates that 
the system satisfies the requirements. 

In this phase, the developers correct any errors uncovered 
by system tests. They also update the draft documentation 
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to reflect the system as it exists when system testing is 
complete. System testing is considered complete when all 
tests specified in the system test plan have been run suc- 
cessfully. 

2.6 ACCEPTANCE TESTING 

During the acceptance testing phase, the system is tested by 
an independent acceptance test team to ensure that the soft- 
ware meets all requirements. Testing by an independent team 
(one that does not have the developers' preconceptions about 
the functioning of the system) provides assurance that the 
system satisfies the intent of the official requirements. 

During this phase, the development team assists the accept- 
ance test team and usually executes the acceptance tests 
under their direction. Any errors uncovered by the accept- 
ance tests are corrected by the development team. Accept- 
ance testing is considered complete when all tests specified 
in the acceptance test plan have been run successfully. 

After the successful completion of acceptance testing, the 
development team delivers final versions of the software and 
the system documentation to the customer and an operational 
readiness review (ORR) is held to evaluate the readiness of 
the system to support operations. 

2.7 MAINTENANCE AND OPERATION 

At the end of acceptance testing, the system becomes the 
responsibility of a maintenance and operation group. This 
marks the beginning of the maintenance and operation phase. 
The nature of the activities during maintenance and opera- 
tion is highly dependent on the type of software involved. 
For most flight dynamics software, this phase typically 
lasts the lifetime of the spacecraft and involves relatively 
few changes to the softv/are. For some support software, 
however, this phase may be much longer and more active as 
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the software is modified to respond to changes in the re- 
quirements and environment. 

The maintenance and operation phase is outside the scope of 
this document. However, enhancements and error corrections 
also progress through a development life cycle but at a much 
lower level of effort than original development. Therefore, 
the recommendations made for original development are, for 
the most part, applicable to development during the mainte- 
nance and operation phase. 
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flECOSySPJJOSyUED SOm'5fAE?E DEVELOPr^EE^T GISIDEUE^SES 


This section presents the SRL's recommended software devel- 
opment guidelines. Each major subsection presents the soft- 
ware development guidelines that are of primary importance 
to the basic development team^ for a particular software 
development life cycle phase. ' Each major subsection 
begins with two summaries. The first summarizes the actions 
and transactions of basic development team members, i.e., 
their major activities, the end products they produce, and 
the methodologies and tools they use for the life cycle's 
primary development activity. The second summarizes special 
actions and transactions of the development team managers, 
i.e., their special activities and the measures they use 
during the life cycle phase. The activities in the manage- 
ment summary are listed in blocks. The first block of ac- 
tivities are those that are specific to the phase and are 
also of interest to higher level managers. Subsequent 
blocks list those activities that art? frequently assumed to 
be goino on or have occurred. ; 

Althourjh the approach is described in terms, of calendar 
phases, it applies to specific development activities when- 
ever they occur. For instance, the methodologies and tools 
recommended for use durinq the detaileil design phase apply 
to detailed dosinn activities when they are performed during 
subsequent life cycle phases. This overlap of activities is 
discussed in Section 2 (see especially Figure 2-1 on 
pace 2-4) - 


The basic development team consists of the customer inter- 
face, who monitors resources and progress; the project man- 
aqor, who serves as technical consultant and manaqes project 
resources; the project leader, who provides technical di- 
rection and day-to-day supervision; and the developers, wlio 
do the tectmical work. See Table C-S on paces C-l(i and C-17 
of Appendix C and Table 1-1 and Fiqure 1-2 of Keference 1. 

The maintenance and operation phase is not addressed in 
this document. 
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Requirements Analysis 


The recommended software development guidelines for the re- 
quirements analysis phase are described in detail below. 

3.1.1 MAJOR ACTIVITIES 

Requirements analysis begins when the requirements defini- 
tion team completes the functional specifications and re- 
quirements document. This document presents the functional 
requirements of the system# including system input and out- 
put, algorithms, and timing and accuracy requirements. The 
development team analyzes the contents of this document for 
completeness, consistency, clarity, and feasibility, and 
then translates the requirements into a form suitable for 
beginning the design. During the requirements analysis 
phase, the team 

o Amplifies and clarifies the functional requirements 
and the algorithms specified to satisfy those re- 
quirements 

© Analyzes the algorithms, mathematical formulations, 
error and stability requirements, and timing and 
accuracy requirements for completeness and feasi- 
bility 

o Determines operational requirements (scenarios) 

o Ensures that all requirements from the mission 

needs statement, the mission problem statement, and 
the end users have been addressed 

o Identifies all external interfaces (both input and 
output) 

o Determines report and display specifications 

o Identifies requirements that are missing or yet to 
be determined (collectively known as TBD require- 
ments ) 

o Identifies any existing software that can be used 
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o Determines computer resource requirements and 
availability 

o Participates in the hardware selection process 

o Communicates findings to the requirements defini- 

tion team 

® Prepares a summary report as the basis for begin- 
ning preliminary design 

© Participates in the system requirements review (SRR) 

During the requirements analysis phase, the development team 
works closely v;ith the requirements definition team, who 
must ansv/er their questions about the requirements and re- 
spond to their requests for requirements changes. Generally, 
the two teams hold requirements review meetings where they 
clarify requirements, discuss problems, and identify items 
needing action. The development team provides the require- 
ments definition team with the results of their analysis for 
incorporation into the final version of the functional spec- 
ifications and requirements document. 


This phase culminates in a final requirements review meeting 
of the requirements definition team and the development team 
and their managers. Maintenance and operation personnel and 
their managers may also be present. The purpose of this 
meeting is to ensure the correctness and completeness of the 
requirements from the viewpoints of all those concerned and 
to identify and assess the impact of any remaining TBD re- 
quirements. Comments and criticisms resulting from this 
meeting are given to the requirements definition team so 
that those issues will be addressed in the final version of 
the functional specifications and requirements document. 

When the final version of the functional specifications and 
requirements document has been completed, an SRR is held to 
evaluate the completeness of the requirements. The SRR is 
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attended by the requirements detinition team and its man- 
agers, the development team and its managers, and others 
involved with the system. See Section A.l of Appendix A for 
details about the SRR. 

Another important function of the requirements analysis 
phase is to produce an initial estimate of the system's size 
and of the schedule and staffing required for development. 
This topic is addressed more fully in Section 3.1.6, 
page 3-13, under "Key Management Activities." 

s 

3.1.2 KND PRODUCTS 

The requirements analysis summary report is the primary 
product. This report summarizes the results of requirements 
analysis and establishes a basis for beginning preliminary 
design. See Section D.3 of Appendix B for information on 
tlie format and contents of the requirements analysis summary 
report. 

3.1.3 MRTHOnOLOGIES 

The recommended methodologies are 
© Project notebook 

o Data collection 

o Librarians 

e Unit development folders 

o Formal rocoivlitui mechanisms for requirements ques- 

tions and clmnges 

o Structuiod analysis for complex data processing 

Those methodolog ies are discussed in tlie Collowinu subsec- 
t ions. 

3 . 1 . 3 . 1 Projoct Notebook 

T!u' project notebook is established and maintaineil by tlie 
development maimgers to provide readily acci?ssibl«> summary 



Requirements Annlys i'lv ‘ ' 

information on the key aspects and phases of the project. 

The notebook is part of the project’s files. The informa- 
tion kept in the project notebook is current; i.e., it is 
updated weekly, biweekly, or monthly depending on the type 
of information. See Section B.2 of Appendix B for informa- 
tion on the format and contents of the project notebook. 

3 . 1 . 3 . 2 Data Col lec t ion 

To understand the development process and to monitor the 
progress of a project, software engineering data must be 
collected throughovjt the development life cycle. The SKL 
recommends that managers make software engineering data col- 
lection a natural byproduct of their managing techniques. 

This topic is treated in more detail in the Sof tware 
Manager's Handbook ( Re f e fence 2 ) . 

3. 1.3. 3 Librar ians 

At GSKC, the librarians are a separate group of personnel 
who are responsible for certain clerical and data entry 
functions. The relationship between the librarians and the 
development team is described in Section 1.4 of Reference 1. 
During a project, the librarians maintain the project library 
(or project notebook) , which is a repository of all project 
information. They also maintain online project libraries, 
enter code, and operate various software tools in support of 
project activities. 

During the requirements analysis phase, tlie librarians 
establisl^ tho project library. In it, t)iey include such 
items as the functional speci f icat ions and requirements doc- 
ument (draft and f ina 1 vers ions as available), questions on 
requ i renuMits and responses t*"* questions, requests for re- 
i^u i i eniiMit chauvies and responses to requests, and the re- 
quirements analysis summary report. In uenoral, the project 
library contains any written material producevi by tlie devel- 
v'>pment team for tlie purpose of recordino ^iecisi^•'ns or 
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communicating information. Necessary management informa- 
tion, such as schedules ahd staffing plans, are also in- 
cluded. Management information produced during the 
requirements analysis phase is discussed in Section 3.1.6, 
page 3-13. 

Librarian functions during this phase are generally limited 
to the operation of software tools, which are discussed in 
Section 3.1.4, page 3-10. 

3.1.3. 4 Unit Development Folders 

The project library is organized according to functional 
units so that all information pertaining to one topic can be 
found in one location in a unit^ development folder. 

During the requirements analysis phase, the most logical 
organization of the materials collected from the development 
team is into the general categories (units) of the system's 
functional requirements (for example, input, output, and 
algorithms) . For the project library to be useful through- 
out the development life cycle, it must be established at 
the beginning of the requirements analysis phase; it must 
contain all material pertinent to the project; and it must 
be logically organized. A description of unit development 
folders is contained in Reference 3, for example. 

3. 1.3. 5 Formal Recording Mechanisms 

As a part of configuration management procedures, tlie de- 
velopment team uses formal recording mechanisms to commu- 
nicate requirements questions and requirements changes. One 


T : — 

A unit is defined by the development manager for con- 
venience; e.g., units may be schedules, system size 
estimates, resource estimates, external interfaces, sub- 
system details, development plans, implementation plans, 
user's guide, and system description. 
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mechanism is the requirements question form. The development 
team uses this form to question the requirements definition 
team about the requirements. Responses to requirements 
questions must be in written form. The requirements defini- 
tion team managers use the form to assign personnel and due 
dates for their team's response to the developers. The 
forms are also used to track TBD requirements. 

Another procedure is used for requirements modifications. A 
request for a requirements modification is made using a re- 
quirements change request form (or engineering change request 
(ECR) ) , on which the requested change and its justification 
are described. Requested changes to the requirements must 
be approved by the manager of the requirements definition 
team and the Configuration Control Board (CCB) . After ap- 
proving a change, the manager adds it to the functional 
specifications and requirements document. 

3. 1.3. 6 Structured Analysis 

No particular methodology is recommended for requirements 
analysis. However, structured analysis is useful for sys- 
tems or subsystems with large quantities of input or output 
data or complex data processing requirements. Structured 
analysis is described in Reference 4, for example. 

3.1.4 TOOLS 

An online configuration management tool is recommended for 
use in configuration management throughout the project. 

During the requirements analysis phase, managers can use 
such a tool to track requirements questions, TBD require- 
ments, and requests for requirements changes. The online 
Configuration Analysis Tool (CAT) program was developed for 
the SEL for use in flight dynamics projects. CAT is doc- 
umented in Reference 5. 
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The SEL recognises the need for an automated requirements 
langusqe,^ but has been unable to identify one that is 
adequate and cost effective for the flight dynamics environ- 
ment. 

3.1.5 MEASURES 

The following subsections describe various measures and 
evaluation criteria for managers to use to assess the re- 
sults of the requirements analysis phase and to determine 
whether enough progress has been made to begin design. 

3. 1.5.1 Objective Measures 

Managers can monitor the progress of requirements analysis 
by examining the number of requirements questions, responses 
to questions, and requirements changes. Several signals 
should alert the manager to problems. For example, a grow- 
ing gap between the number of questions submitted and the 
number of responses received or a large number of require- 
ments changes due to errors may indicate problems with the 
clarity, correctness, or completeness of the requirements as 
presented in the functional specifications and requirements 
document. Managers can use data from similar past projects 
to assess the meaning of the relative sizes of these numbers 

The number of TBD requirements is the most important measure 
to be examined during this phase, since unresolved TBD re- 
quirements can necessitate severe design changes later in 
the project. The TBD requirements must be categorized ac- 
cording to their severity. TBD requirements concerning ex- 
ternal interfaces are the most critical, especially if they 
involve system input. Internal algorithms are generally not 


^Examples of requirements lanauaqes include Problem 
Statement Language/Problem Statement Analyzer (PSL/PSA) 
(Reference 6) and Multi-Level Exoression Design Language - 
Requirements Level (MEDL-R) (Reference 7). 
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as severe, unless they concern data processing reguirements. 
Output requirements are, in general, not as severe unless 
they concern data being transmitted to other systems. 

A TDD requirement is considered severe if it could affect 
the functional design of one or more subsystems or of the 
high-level data structures needed to support the data proc- 
essing algorithms. Preliminary design should not proceed 
until all severe TDD requirements have been resolved. A TDD 
requirement is considered nominal if it affects a portion of 
a subsystem involving more than one module. Preliminary 
design can proceed unless large numbers of TDD requirements 
exist in one functional area (for example, more than 5). 
However, these TDD requirements must be resolved during 
preliminary design. An incidental TDD requirement is one 
that affects only the internals of one module, incidental 
TDD requirements must be resolved by the end of detailed 
design. 

For each TDD requirement, managers must estimate the effect 
on system size, required effort, cost, and schedule. Often 
the information necessary to resolve a TDD requirement is 
not available until later, and design must begin to meet 
fixed deadlines. These estimates will help predict the 
uncertainty in the development schedule due to unresolved 
TDD requirements. 

.1 . 1 • 5 . 2 Fvaluation Criteria 

To determine whether or not the development team is ready to 
proceed with preliminary design, managers must consider the 
following questions: 

o Is all external input and output completely defined? 

o Have all TDD requirements been identified and their 
impact assessed? 
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o Are all necessary algorithms identified? Are the 
identified algorithms complete and correct? Are 
they optimal? Are error and stability limits 
defined? 

© Are the environmental constraints (for example, 
timing, memory, and accuracy) clear? 

o Are the requirements feasible, given the environ- 
mental constraints? Are sufficient computer 
resources available? 

o Are the requirements traceable? Does the func- 
tional specifications and requirements document 
provide a basis for defining acceptance tests? 

3.1.6 KEY MANAGEMENT ACTIVITIES 

Planning is the manager's primary activity during the re- 
quirements analysis phase. During this phase, the develop- 
ment team, managers produce the softv.'are development plan. 
Toward the end of the phase, the transition to the prelim- 
inary design phase must be planned. These planning activ- 
ities are in addition to other activities such as monitoring 
progress, ensuring cooperation among all groups involved, 
and reviewing the results of requirements analysis. These 
key activities are discussed in further detail in the fol- 
lowing subsections. 

3 . 1 . 6 . 1 Software Development Plan 

The software development plan contains specific information 
about the technical and management approaches of the current 
project throughout its life cycle. The development team 
managers prepare this document during the requirements anal- 
ysis phase. Because of the primary importance of this plan, 
it is described in great detail in the Software Manager's 
Handbook (Reference 2) . 
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The SEL often uses the flight dynamics software development 
projects as experiments in software engineering research. 
Thus, frequently, a specific software engineering approach 
is applied to a project to evaluate its effectiveness. 
Therefore, data collection procedures and details concerning 
the application of the specified approach must be estab- 
lished at the beginning of the project life cycle. 

3 . 1 . 6 . 2 Resource and Cost Estimates 

At the beginning of this phase, the managers must estimate 
the amount of code to be developed and the effort required 
to develop it. Sufficient information is not usually avail- 
able to use a sophisticated resource estimation model, but a 
rough model can be applied. Historical knowledge of similar 
systems can be used to estimate the size of the system, and 
historical productivity figures can be used to estimate the 
amount of effort required. From these estimates, the amount 
of time and effort necessary for each life cycle phase is 
allocated, and the first cost figures and schedules are 
produced. ! 

3. 1.6.3 Phase Transition Plans 

Toward the end of the requirements analysis phase, managers 
must plan an orderly transition to the preliminary design 
phase. They must convey to the development team members the 
parts of the software development plan that apply to pre- 
liminary design (for example, design standards and con- 
figuration management procedures) and instruct them in the 
specific software engineering approach to use during design. 
They must ensure that the development team is trained in 
design technologies at the beginning of the preliminary 
design phase. 
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3 . 1 . G . 4 Other Management Functions 

During the requirements analysis phase, the managers must 
also 

e Monitor adherence to planned schedules and resource 
expenditures. 

o Ensure adherence to data collection, quality as- 
surance, and configuration management procedures. 

G Review the results of the requirements analysis 
process. 

o Ensure cooperation among the various groups in- 
volved (that is, development team, requirements 
definition team, user organization, and librarians) . 

o Schedule and participate in the requirements anal- 
ysis review and ensure that all pertinent groups 
participate . 

o Ensure that all facets of the project are com- 
pletely visible (that is, know exactly where the 
project is and where it is going at all times) . 
Project visibility is critically important. Man- 
agers must know at all times the exact status of 
all task activities and the detailed plans for de- 
velopment completion. This is necessary so that 
problems can be dealt with when they occur rather 
than later in the process, when their impact is 
likely to be greater. 

© Participate in the £RR. 

The Software Manager's Handbook (Reference 2) presents 
further inf'^rmation about the manager's activities through- 
out the development life cycle. 
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Preliminary Design 

o Defines all interf,&ces between subsystems--that is, 
completely specifies all input and output for each 
subsystem, including 

Data set layouts (record content and file 
structure) 

- Data transferred in memory 

- User input 

Screen, printer, and plotter output 

® Refines the subsystem design to two levels below 
the subsystem drivers, including preparation of 
functional baseline diagrams (treecharts) through 
two levels belov; subsystem drivers and module 
Prologs and Process Design Language (PDL) through 
one level below subsystem drivers (see Figure 2-2 
on page 2-6) 

\ ; 

© Determines error processing and recovery strategy, 
especially with respect to handling system input/ 
output errors 

o Resolves as many remaining 'I'UD requirements as pos- 
sible; assesses the impact of those not resolved 

© Examines all requirements to ensure that they are 
met by the functional capabilities of the subsys- 
tems defined in the preliminary design 

o Identifies all existing software to be used in the 
system 

© Prepares preliminary design documentation as a 
basis for the preliminary design review (PDR) 

o Participates in the PDR and then incorporates 

changes recommended at the PDR into the preliminary 
des ign 
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The preliminary design phase culminates in the PDR, attended 
by the development team and its managers, the requirements 
definition team and its managers, and others involved with 
the system. At the PDR, the development team presents the 
functional design of the system and the rationale for choos- 
ing that design over alternatives. The presentation is 
based on the preliminary design documentation and may re- 
quire a series of meetings if the system is large. See Sec- 
tion A. 2 of Appendix A for details about the PDR. 

For the PDR presentation, the participants evaluate the 
functional design of the system for completeness and cor- 
rectness and give comments and criticisms to the development 
team during and immediately after the presentation. The 
preliminary design is complete when the development team has 
adjusted the preliminary design documentation to respond to 
comments and criticism expressed at the PDR. 

3.2.2 END PRODUCTS 

The preliminary design report is the primary product. It 
presents the functional design of the system and forms the 
basis for the detailed design document produced during the 
next life cycle phase. See Section B.4 of Appendix B for 
the format and contents of the preliminary design report. 

3.2.3 METHODOLOGIES 

The SEL recommends the following methodologies for use 
during the preliminary design phase; 

c Project notebook 

0 Data collection 

o Librarians 

o Unit development folders 

o Formal recording of design decisions and changes 

o Configuration management procedures 

o Design walkthroughs 
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o Iterative refinement 

o Information hiding and data abstraction 

o PDL 

Data collection and maintenance of the project notebook must 
continue throughout the development life cycle. The remain- 
ing methodologies for the preliminary design phase are elab- 
orated in the following subsections. 

3 . 2 . 3 . 1 Librarians and Unit Development Folders 

The librarians continue to maintain the project library. 

They add to the library such items as design decision notes, 
design change forms, and all preliminary design documenta- 
tion, as well as pertinent management materials. (The man- 
agement information produced during the preliminary design 
phase is discussed in Section 3.2.6, page 3-28). Since re- 
quirements analysis continues throughout the development 
life cycle, the development . team may produce more require- 
ments questions and requests for requirements changes during 
this phase. The librarians also add these inquiries and 
their responses to the project library. 

During this phase, the development team managers organize 
the project library materials in the unit development 
folders by major subsystems and by functional areas within 
each subsystem. This organization corresponds to the first 
level below the subsystem drivers on the functional baseline 
diagrams (see Figure 2-2 on page 2-6) . 

The librarians enter module prologs^ and PDL as well as 
operate CAT, which is discussed in Section 3.2.4, page 3-25. 


^Module comments describing the module's purpose, opera- 
tion, calling sequence arguments, external references, etc. 
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■3 . 2 . 3 . 2 Formal Recording Mechanisms and Confirm rat ion Man- 
asimnent Procedures ■ 

As part of the configuration management of the project, the 
developers use formal recording mechanisms to document 
design decisions and changes. When a design decision is 
made, it is recorded as a design decision note. Because 
these notes document the design process--particularly the 
evaluation and selection of alternatives--they are a valu- 
able reference for the developers througliout the software 
life cycle. 

Once design decisions have been finalized by the development 
team managers, design change forms are used to record 
furtlier changes. The use of formal design change procedures 
enables the team managers to control design changes and to 
ensure that all team members are kept informed of the cur- 
rent state of the design. 

The use of formal recording mechanisms for particular life 
cycle activities must continue throughout the life cycle 
wlienever those activities occur. For instance, requirements 
questions forms and requirements change request forms con- 
tinue to be used after the requirements analysis phase for 
requests for clarification or changes to the requirements. 

The procedures for conf iguration management must be strictly 
adhered to throughout tlie development life cycle. These 
procedures specify the forms to be used for recording vari- 
ous inquiries, decisions, and requests for changes and 
address the processing of such forms (for example, responsi- 
bility for response, authority for approval, and distribu- 
tion). Strict procedures are especially important in the 
area of chansie control. 

Conf igurat ion management procevlures also specify the control 
of online project libraries. Tlie procedures must specify 
the point at which modules are moved from the individual 
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developer's jurisdiction and placed in the project libraries. 
At that time, the modules are placed under configuration 
control. Any changes are performed by a specially desig- 
nated person or group of people (usually the librarians) and 
must be recorded formally by means of a change report form. 
These procedures for the control of online libraries apply 
to the preliminary design phase only if a decision is made 
to place module prologs and PDL under configuration manage- 
ment. 

3. 2. 3. 3 Design Walkthroughs 

Design walkthroughs are held throughout the preliminary 
design phase by development team personnel and their man- 
agers to review design elements and to identify problem 
areas and TBD requirements. This peer review is an impor- 
tant quality assurance procedure. After design walk- 
throughs, managers assign personnel to resolve problems and 
schedule their response as part of configuration manage- 
ment. The development team leader records any decisions 
made at a design walkthrou^. in oesign decision notes. 

Design walkthroughs are also used to identify the point at 
which design elements are placed under configuration control 
by the development team manager. A design element is usu- 
ally put under configuration control when it has been incor- 
porated in the preliminary design documentation or presented 
in a design walkthrough or at the PDR, depending on the sta- 
tus of the design. After that, changes to the oesign must 
be maoe according to the change control procedures and must 
be recorded by means of design change forms. A description 
of design walkthroughs is contained in Reference 8, for ex- 
ample . 

3. 2. 3. 4 Design Techno log ies 

The SDL recommenus iterative refinement i for example. Refer- 
ence '•)) as the primary methoo for producing the system 
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' design. When a substantial amount of existing design and/or 
code is to be reused (for example, 20 percent or more) , it- 
erative refinement is recommended to functionally partition 
the system into modules. This process is preferred to 
strict successive refinement (used in top-down design) when 
adapting the design to the structure of the existing design 
and code. 

In the functional partitioning process, the SEL recommends 
the principles of information hiding (for example. Refer- 
ence 10) and data abstraction (for example. Reference 11) . 

An example of these techniques is the use of common inter- 
face routines for performing input or output operations so 
that the format and structure of each external data set are 
known to only one routine and are transparent to the rest of 
the system. ; 

3.2. 3. 5 Process Design Language (Program Design Language) 

The SEL highly recommends the use of PDL during design as a 
very beneficial and cost-effective methodology. Comparable 
to the blueprint in hardware, PDL communicates the concept 
of the software design in all necessary detail. It provides 
a complete, formal, algorithmic specification for a software 
component. Its use enables the designer to communicate the 
exact intent of the design and thus reduces errors due to 
misinterpretation of the design by reviewers and coders. A 
description of PDL is contained in Reference 12. 

3.2.4 TOOLS 

An online configuration management tool (for example, CAT) 
is used throughout the development life cycle. In this 
phase, it can be used to maintain detailed schedule informa- 
tion at the subsystem and module levels. 
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One new tool is recommended: 

o An automated PPL processor (for example, Refer- 
ence 13) (if one is available) . This tool enforces 
consistency of PDL usage among development team 
members and also performs syntax-checking opera- 
tions. 

3.2.5 MEASURES 

The following subsections present various measures and eval- 
uation criteria for managers to use in assessing the prelim- 
inary design phase. 

3.2.5. 1 Objective Measures 

During this phase, managers continue to use the same objec- 
tive measures as during requirements analysis. In partic- 
ular, they monitor 

o Number of requirements questions, responses to 
questions, and requirements changes . The number of design 
changes must also be examined. Numerous design changes not 
attributable to requirement changes should alert the manager 
to problems; these changes may indicate that the development 
team does not really understand the requirements. 

© Number of TED requirements (see Section 3. 1.5.1, 
page 3-11) . Managers must assess how each TBD requirement 
will affect system size, required effort, cost, and sched- 
ule. By the end of this phase, only incidental TBD re- 
quirements can be left unresolved. 

o Number of interfaces . The number of interfaces per 
subsystem is an indication of that subsystem's complexity: 
a subsystem with a large number of interfaces relative to 
its size will require more time for implementation and 
thorough testing. Data from past projects of a similar 
nature can be used to interpret the relative sizes of these 
numbers. 
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To help monitor the, progress of the preliminary design, man- 
agers can produce and'use ' 

© A detailed checklist of design formalisms to be 
produced . Because preliminary design documentation contains 
all design formalisms produced during preliminary design, 
all items on the checklist must be completed before the PDR. 

3.2.5. 2 Evaluation Criteria 

To evaluate the correctness and completeness of the prelim- 
inary design and determine whether the development team is 
ready to proceed with detailed design, managers must con- 
sider the following questions; 

o Have all requirements been mapped into functional 
capabilities of specific subsystems? 

o Have alternative design approaches been examined 
and rationally discounted, and has the simplest 
design been chosen? 

o Is the partition into subsystems sensible? Are 
functions and capabilities allocated logically? 

Does the design minimize the transfer of control 
information? Does the design have low coupling 
between subsystems and high cohesion within each 
subsystem? (Coupling and cohesion are defined in 
Reference 14, for example.) 

® Are all interface descriptions complete at both the 
system and subsystem level? 

e Are the data set layouts for all external data sets 
completely specified? 

o Are the required baseline diagrams and module pro- 
logs and PDL provided to a sufficient level of 
detail? 
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. © Is the error handling and recovery strategy 

comprehensive? 

o Is the estimate of resources adequate and the 

schedule reasonable? Has time been allocated for 
contingencies, training, and the like? 

a Has the impact of any remaining TBD requirements 
been assessed? 

3.2.6 KEY MANAGEMENT ACTIVITIES 

During this phase, the managers' focus begins to change from 
planning to monitoring. Specifically, managers 

a Provide required training for the development team 

o Ensure adherence to 

Design standards 

0 

Configuration management procedures 
- Reporting procedures 

Data collection procedures 
Quality' assurance procedures 

o Review the design produced, participate in design 
walkthroughs, and resolve" TDD requirements 

o Monitor adherence to planned schedule and expendi- 
ture of resources, and update cost and resource 
estimates and schedules 

9 Ensure that all facets of the project are com- 
pletely visible 

o Coordinate communication between the development 
team and the other groups with which they must . 
interact (for example, the librarians and the re- 
quirements definition team) 

o Plan transition to the detailed design phase 
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o Schedule and participate in the PDR, and ensure 
that all pertinent groups participate 

Further details on the refinement of resource and cost esti- 
mates and on phase transition planning are discussed in the 
following subsections. 

3. 2. 6.1 Resource and Cost Estimates 

During the preliminary design phase, managers must monitor 
the development team's adherence to cost and resource esti- 
mates and the schedules in the software development plan 
(see Reference 2) . The percentages of effort and time ac- 
tually expended versus the percentages of the quantities 
planned to be expended in terms of the work accomplished are 
good measures to examine for monitoring progress. 

By the end of this phase, the managers can refine and update 
resource and cost estimates made during the requirements 
analysis phase. System size is better known, as are re- 
sources expended and progress made to date. Enough in- 
formation is usually available to use a formal resource 
estimation model. It is important for the manager to use a 
model that is tuned to the specific environment and cor- 
responds well with the resources expended for similar past 
projects. The Meta-Model has been developed using SEL data 
(see Reference 15) . However, managers must never completely 
rely on any formal resource estimation model. Rather, they 
must use the results of the model, together with historical 
knowledge of similar systems, to update resource and cost 
estimates. The new estimates are more accurate because they 
are based on additional information and model support. 

From these new estimates, managers prepare schedules and 
staffing plans. Schedules are refined to reflect the sub- 
system division established in the preliminary design. The 
managers add these new estimates and schedules to the soft- 
ware development plan to form the basis for monitoring 
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progress during the next/life cycle phase. The process of 
monitoring actual progress versus planned progress and 
updating the plan as more detailed information becomes 
available continues throughout the project )ife cycle. 

3. 2. 6. 2 Phase Transition Plans 

Toward the end of the preliminary design phase, managers 
must plan the transition to the detailed design phase; i.e., 
they must plan the detailed design phase so chat ma.ior sub- 
systems are designed concurrently. In addition, they must 
determine the staffing levels and assignments necessary to 
perform the detailed design. Managers usually add persontiel 
to the development team at the beginning of detailed design. 
They must ensure that these personnel receive any training 
required and that new members are informed of work assign- 
ments, design standards, softv/are engineering approaches, 
and quality assurance and configuration management proce- 
dures. The online libraries must also be established to 
store module prologs, PDL, and reused code during detailed 
design. 
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Tlie recommended software development ouidelines for the de- 
tailed desion pliase are described in detail below. 

3.3.1 MAJOR ACTIVITIES 

Detailed desiqn beqins after the PDR unless comments and 
criticism expressed at the PDR indicate serious fjroblems or 
«ief iciencios with the preliminary desiqn. Durinq detailed 
desiqn, the team elaborates the system architecture defined 
by the preliminary desiqn to the subroutine level. The 
detailed desiqn process is an extension of the activities 
beaun durinq preliminary desinn until "code-to" specifica- 
tions are complete. Speci f ica I ly , the team 

o Sviccess ivelv' refines each subsystem until each com- 
ponent performs a sinolo function and can be coded 
as a si nolo module 

0 Prepares detailed baseline diaqrams (treecharts) to 
tlie subroutine level 

o Finishes specifyino detailed formats of all system 
and subsystem input and output 

o Completes p'roloos and PPL for all modules 

o Specifies COMMON blocks and internal interfaces 
between modules 

o Specifies the staoed implementation plan, includinq 
capabilities to be included in each bu i Id./’release 
and the detail(>d milestone schedule for oacli build'' 
release 

o Prepares detailed desion documenta t ion as a basis 
for the critical desiqn review (CDR) 

o Part icipat(‘S in the COR anil incorporates ctianoes 
recommended at the CDR into the detailed desion 
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The detailed design documentation contains all design for- 
malisms and must be distributed to everyone attending the 
CDR before the CDi{ meetings. The design formalisms must be 
prepared in accordance with the guidelines and standards 
specified (that is, size, complexity, functionality of mod- 
ules, Prolog contents, and PDL usage) . 

The detailed design phase culminates in tlie CDR, attended by 
the development team and its managers, the requirements def- 
inition team and its managers, and others involved with the 
system. At the CDR, the development team presents the de- 
tailed design of each subsystem for critical review. This 
presentation is based on the detailed design documentation 
and may require a series of meetings if the system is 
large. See Section A. 3 of Appendix A for details about the 
CDR. 

For the CDR presentation, the participants evaluate the de- 
tailed design of the system to determine wliether the design 
is correct and complete enough to begin implementation. 

They also review build/release capabilities and schedule for 
feasibility. The detailed design is complete when the de- 
velopment team has adjusted the detailed design to respond 
to comments and criticism expressed at the CDR. 

J.3.2 KND PRODUCTS 

The detailed design document is the primary product. Tliis 
document is an extension of the preliminary design report. 
See Section U.5 of Appendix U for the format and contents of 
the detailed design document. 

3.J.J MCTHODOLOGIUS 

Uecause the activities of detailed design are an extension 
of those performed during preliminary design, the same 


3-jg 



Detailed Design 


methodologies are used (see Section 3.2.3, page 3-21). They 
are repeated below: 

© Project notebook 

o Data collection 

o Formal recording of design decisions and changes 
o Configuration management procedures 
o Design walkthroughs 

o Iterative refinement 

0 Information hiding and data abstraction 

o PDL 

New activities for this phase are described below: 

© Librarians . The librarians begin to transfer ex- 

isting code to be used in the implementation into the proj- 
ect's online source code libraries. They continue their 
activities of preliminary design, including adding all ma- 
terials produced during the detailed design phase to the 
project library. The organization into unit development 
folders according to subsystem (started during preliminary 
design) is continued and refined during detailed design. 

© Unit development folders ., A chart is added to the 
unit development folder for each subsystem, showing each 
module in the subsystem and the planned and actual starting 
and ending dates for each of the major phases (that is, de- 
sign, code, and test) for the module. The librarians update 
these charts to reflect current development status for each 
module throughout the remainder of the project life cycle. 

3.3.4 TOOLS 

The same tools recommended for use in the preceding phases 
are used (see Section 3.2.4, page 3-25) : 

o An online configuration management tooI--for ex- 
ample , CAT 

e An automated PDL processor, if one is available 
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A new tool for detailed design is 

o An online source code library management system , 
which is to be used to manage the project libraries. Such a 
tool, if available, is an important part of the configura- 
tion management procedures because it can be used to enforce 
strict change control procedures on the project libraries 
containing PDL and source code that have been placed under 
configuration control. 

3.3.5 MEASURES 

The measures and evaluation criteria used during detailed 
design are similar to those used for preliminary design. 
Further explanation is given in the following subsections. 

3. 3. 5.1 Objective Measures 

As specified for preliminary design (see Section 3.2. 5.1, 
page 3-26) , managers monitor the following objective meas- 
ures, repeated below: 

e Number of requirements questions. 

o Number of responses to requirements questions, 

© Number of requirements changes. 

© Number of design changes. 

o Number of interfaces. 

o Number of T3D requirements . The number of TBD re- 
quirements is the most important quantity to be examined. 
Ideally, all TBD requirements must be resolved by the end of 
this phase. If this goal is impossible to achieve, the man- 
agers must assess how the remaining TBD requirements will 
affect system size, required effort, cost, and schedule. 

o A detailed checklist of all design formalisms . 

This list can be used to' evaluate the design's completeness. 
Because the detailed design documentation contains all the 
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design formalisms produced for detailed design, all items on 
the checklist must be completed 'before the CDR. 

One new measure can be used by the managers to monitor 
progress: 

o An updated estimate of the number of lines of code 
in the system . By the end of detailed design, managers know 
the projected num.ber of modules in the system. The budgeted 
effort rate can then be examined by computing the number of 
lines of code per (budgeted) effort unit and the number of 
modules per (budgeted) effort unit. Managers then can com- 
pare these figures with the same figures for similar past 
projects to determine whether or not enough effort has been 
budgeted to complete development. 

3 . 3 . 5 . 2 Evaluation Criteria 

To evaluate the correctness and completeness of the design 
and to determine whether the development team is ready to 
proceed with implementation, managers must consider the fol- 
lowing questions: i 

I 

e Have all items on the checklist of required design 
formalisms been completed? For example, are all 
external data sets completely defined and all base- 
line diagrams (treecharts) provided to the subrou- 
tine level? 

9 Is the design correct? Will the transformations 

specified produce the correct output from the input? 

o Is the design robust? Is user input examined for 
potential errors before processing continues? 

o Is the design testable? 

o Have all design guidelines and standards specified 
been followed? 
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o Are the descriptions of each component clear enough 
and sufficiently unambiguous so that implementors 
can proceed autonomously? 

o Have all TBD requirements been resolved? If not, 

how will the remaining TBD requirements affect sys- 
tem size, required effort, cost, and schedule? 

o Is the build/release schedule structured to provide 
early testing of end-to-end system capabilities? 

Is the schedule reasonable and feasible for imple- 
menting the design? 

o Is the estimate of resources adequate for complet- 
ing development? 

Managers can evaluate the quality of the design by consider- 
ing the following factors; 

o The level of information hiding (that is, how well 
have data usage and access been localized. Are 
modules secretive in the way in which they perform 
their functions?) 

o The degree of coupling between modules (that is, 
intramodule dependencies are minimized) 

o The cohesiveness of the lowest level components 
(that is, each module has a single purpose) 

3.3.6 KEY MANAGEMENT ACTIVITIES 

During this phase, the manager's concerns are identical to 
those for preliminary design (see Section 3.2.6, page 3-28) 
and are repeated below. The activities include both plan- 
ning and monitoring. Specifically, the managers 

o Ensure adherence to 

Design standards 

Configuration management procedures, especially 
change control 
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o Review the design produced, participate in .esign 
walkthroughs, and resolve TBD requirements 

© Monitor adherence to planned schedules and expend- 
iture of resources, and update cost and resource 
estimates and schedules 

o Ensure that all facets of the project are 
completely visible 

o Ensure cooperation between the development team and 
the other groups with which thev must interact 

o Plan transition to the implementation phase 

o Schedule and participate in the CDR, and ensure 
that all pertinent groups participate 

For the transition to the implementation phase, it is usu- 
ally necessary to increase the size of the development team 
substantially to handle the simultaneous implementation of 
the builds for each subsystem. Managers must inform the 
development team of the software engineering approaches to 
be used during implementation and must provide required 
training. Also, the members of the development team must 
understand the code and testino standards, the quality as- 
surance procedures, and the configuration manaoement proce- 
dures to be followed in addition to their individual areas 
of responsibility. 

Managers must also ensure that the online project libraries 
are established, that the strict change control procedures 
concerninq tliese libraries are followed, and that the job 
control language (JCL) for building and testing the system 
is prepared for the developers so that they can start imple- 
mentation immediately after the CDR. 
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The recommended software development guidelines for the 
implementation phase. are described in detail below. 

3.4.1 MAJOR ACTIVITIES 

Implementation begins after the CDR unless comments and 
criticism expressed at the CDR indicate serious problems or 
deficiencies with the detailed design. In implementation, 
the development team 

© Completes preparation of JCL and command procedures 
necessary to build and test the system 

o Codes new modules from the detailed design specifi- 
cations and revises old. routines required to meet 
the requirements 

o Integrates new modules into the growing system or 
subsystem 

e Prepares data for performing unit/integration and 
release testing 

© Performs unit/integration testing to ensure that 
newly added capabilities function properly 

o Prepares test plans for each build/release 

o Executes tests specified by the test plan for each 
build/release and reviews test results 

o Prepares the system test plan for use during the 
system integration and testing phase 

o Prepares drafts of the user's guide and system 

description documents, based on the material in the 
detailed design document 

The system is implemented according to the staged implemen- 
tation plan prepared by the developers during the detailed 
desinn ohase. For each release, individual developers code 
and test the modules identified as belonging to a particular 
build of each subsystem. At the same time, members of the 
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development team prepare the test plan for the release com- 
prising the builds under development. This test plan is 
designed to test the functional capabilities of the release 
and is reviewed for correctness and completeness by develop- 
ment team members and their managers. 

When the developers have completed all coding and unit test- 
ing for the release, they rebuild the system from source 
code and execute the tests specified in the release test 
plan. The development team and its managers carefully re- 
view test results to identify discrepancies. 

For each release, the test plan evaluates the functional 
capabilities of the release as it is defined in the staged 
implementation plan. A sampling of tests from previous 
releases, called regression tests, is included in each test 
plan to ensure that the newly added capabilities have not 
affected the functioning of the previously implemented 
capabilities. During implementation of the last release, 
the development team prepares the system test plan in addi- 
tion to the test plan for the last release. The system test 
plan is the basis for system testing performed during the 
next life cycle phase. It is designed to test the func- 
tional capabilities of the system as specified in the 
requirements documentation. 

An independent acceptance test team prepares the acceptance 
test plan based on the information in the functional spec- 
ifications and requirements document. The acceptance test 
team usually consists of analysts who will use the system. 
This team frequently includes members of the organization 
that prepared the functional specifications and requirements 
document. 


3-44 



Implementation 

3.4.2 END PRODUCTS 

During implementation, the development team produces the 
following products: 

© Completed code for the system 

© Supporting files necessary for building and execut- 
ing the system (for example, JCL, command proce- 
dures, and load modules) 

© Test plans and results for each build/relea^e 

o System test plan 

o Draft user's guide 

o Draft system description 

The test plans are generally produced as informal docu- 
ments. Each one contains a set of tests designed to test 
the functional capabilities of a particular release or of 
the entire system. See Section B.6 of Appendix B for the 
format and contents of test plans. 

The user's guide and the system description may be produced 
as two separate documents or combined into one. During this 
phase, this material is prepared in draft form. Most of the 
information needed is already available in the detailed de- 
sign document. See Sections B.7 and E.8 of Appendix B for 
the format and contents of the user’s guide and system de- 
scription . 

An independent acceptance test team produces the acceptance 
test plan. 

1.4.3 METHOD DLOGIES 

The SEL recommends the following methodologies for impienen- 
tation . 

© Project notebook 

o Data collection 
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Inipleraentation , . , 

o Librarians ‘ , 

o Unit development folders 

o Formal recording of changes 

o Configuration management procedures 

Data collection and maintenance of the project notebook con- 
tinue as is recommended in the preceding life cycle phases. 
New applications of the others are described in Sec- 
tions 3. 4. 3.1 and 3. 4. 3. 2 below. 

In addition, the following new methodologies are also used: 

o Coding standards 

o Structured code 

© Code reading 

o Top-down implementation 

e Builds/releases 

© Functional (thread) testing 

o Formal test plans , ' 

I 

The remaining methodologies are described in more detail in 
Sections 3. 4. 3. 3 through 3. 4. 3. 6 below. 

3 . 4 . 3 . 1 Librarians and Unit Development Folders 

During implementation, the librarians support tlie develop- 
ment team by entering newly developed code, entering modifi- 
cations for reusable code, and operating the software tools 
discussed in Section 3.4.4 below. The librarians also up- 
date the project's permanent source code libraries, incorpo- 
rating changes made to the source code after it has been 
placed under configuration control. In this function, the 
librarians become an important part of the configuration 
management procedure. 

The librarians maintain the central project library and keep 
it organized into unit development folders by subsystem. 

They add all materials produced during implementation to the 
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project library; test plans and results for each build/ 
release, and drafts of user's guide and system description 
information. They also add change reports for changes made 
to any parts of the system that are under configuration con- 
trol (for example, the functional specifications and re- 
quirements document, the detailed design document, and the 
project's permanent source code libraries). The librarians 
also update the charts (started during detailed design) that 
show the exact status of each module in the system. 

3.4. 3. 2 Formal Recording Mechanisms and Configuration 
Management Procedures 

Configuration management procedures must be strictly adhered 
to during this phase. Source code for a module is placed 
under configuration control when the individual developer 
has coded, compiled, and tested the module successfully. At 
that point, the module is moved from the developer's juris- 
diction into a permanent project source code library. Any 
further changes to the module must be approved by the devel- 
opment team leader before they are made by the librarians. 
These changes must be recorded on development change report 
forms. 

In'addition, the formal recording mechanisms used in the 
preceding life cycle phases for requirements questions and 
changes, and design decisions and changes, are used for re- 
quirements analysis and design activities that occur during 
implementation. 

3. 4. 3. 3 Structured Code and Coding Standards 

The SEL recommends use of structured code (that is, using 
only the basic structured constructs) in implementing the 
design of the modules. These constructs correspond to those 
in the module’s PDL. The principles of structured program- 
ming are described in Reference 16, for example. 
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The code must conform to the coding standards specified. 
Quality assurance procedures must be enforced by the man- 
agers to ensure that the developers adhere to those stand- 
ards. 

3.4. 3. 4 Code Reading 

After a developer codes and successfully compiles a module, 
another member of the development team reads the code to 
verify that it performs the functions specified in the de- 
sign and to check for common coding errors. The reader must 
review and return the code within half a day so that the 
developer is not delayed. Code reading identifies errors in 
the implementation of the design before testing begins. 

This review procedure is usually adequate. Occasionally, 
however, the development team may hold more formal walk- 
throughs for high-level or very complex modules, but this is 
unnecessary for most modules. Details on code walkthroughs 
are contained in Reference 17, for example. 

3. 4. 3. 5 Implementation Technologies 

Implementation proceeds according to the builds and releases 
defined during detailed design in the staged implementation 
plan. A build is a portion of a subsystem that performs 
certain designated functions; a release is a portion of the 
system, composed of one or mote builds, that has certain 
end-to-end functional capabilities. The modules in each 
subsystem build and the builds in each release are specified 
in the staged implementation plan. 

Each subsystem build is implemented in a top-down fashion: 
i.e., if the baseline diagram is pictured as a map with- 
North at the top, modules in the build are coded and tested 
in the order in which they appear in a nor thwest-to- 
southeast sweep of the baseline diagram (from the highest 
level to the lowest level and simultaneously from left to 
right) . Developers test modules by integrating them into 
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the ijrowing subsystem and using the existing, previously 
tested subsystem as a test bed. Modules not yet implemented 
exist in the subsystem as stubs (that is, fully executable 
modules containing no executable instructions except to 
write a message that the module was entered and has returned 
control to the calling module) . 

Top-down implementation tests both the module's integration 
into the growing subsystem and its internal code. It also 
exercises the higher level and data input modules more fully 
and eliminates building test drivers that themselves require 
testing. Some modules may require unit testing in an iso- 
lated environment before they are integrated into the sub- 
system, but this should be necessary only in special cases 
(for example, to verify a particular algorithm). 

3. 4. 3. 6 Functional Testing and Formal Tost Plans 

After the builds of a particular release are completed and 
integrated into the system, the release's end-to-end proc- 
essing capabilities (called "threads") are tested by the 
developers. An important part of this functional testing 
process is the formal test plan, wliich specifies the func- 
tional capabilities to be tested and the criteria for deter- 
mining wliether or not the test is successful. This is done 
for eacli test in the release test plan. The use of a formal 
test plan thus allows release testing to proceed in a logi- 
cally organized manner and facilitates agreement among man- 
agers and developers as to when release testing is 
satisfactor ily completed. The system test plan, prepared 
during the implementation phase, serves the same purpose 
during the system integration and testing phase that fol- 
lows. Testing is described in Reference IB, f-.r example. 



Implementation 


3.4.4 TOOLS 

The development team uses 

e An online configuration management tool (for ex- 
ample, CAT) . The tool is important in configuration manage- 
ment of the project's permanent source code libraries to 
track development changes. It is also used to maintain the 
detailed schedule for the development of each module in the 
system. In this phase it is very useful for maintaining 
information about discrepancies identified during testing. 
During testing of each release, discrepancies between how 
the system works and how it is supposed to work are identi- 
fied. For large systems, the number of discrepancies that 
must be rectified can be substantial. Managers must keep 
track of these discrepancies, assign personnel to resolve 
them, set dates for resolution, and verify that the discrep- 
ancies have been resolved. A tool such as CAT makes this 
task easier. 

o An online source code library management system . 

o A structured FORTRAN preprocessor . This tool, 
which translates structured constructs into valid FORTRAN 
code, allows the programmer direct use of the standard 
structured constructs and thus facilitates structured pro- 
gramming.' A structured preprocessor (SFORT) (Reference 19) 
is available in the SEL environment. Some versions of 
FORTRAN (for example, those conforming to the FORTRAN 77 
language standards) already contain the structured con- 
structs as part of the language and therefore do not require 
the use of a structured preprocessor to provide those capa-'^ 
bilities . 

3.4.5 MEASURES 

The following subsections present various measures and eval- 
uation criteria that may be used to assess the implementa- 
tion phase. 
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: 3 . 4 . 5 . 1 Objective Measures 

As in preceding life cycle phases, managers monitor the num- 
ber of requirements questions, responses to requirements 
questions, requirements changes, and design changes. Man- 
agers also ensure that all TED requirements are resolved by 
the beginning of the implementation phase. If this is not 
possible, managers must reassess how remaining TED require- 
ments will affect system size, required effort, cost, and 
schedule . 

Managers must also monitor the following additional objec- 
tive measures during the implementation phase: 

o Productivity rates (number of lines of code, number 
of modules, and number of pages of documentation per effort 
unit) . As implementation progresses, managers can obtain 
more accurate estimates of the number of lines of code and 
number of modules. Then they can update estimates of the 
budgeted productivity or effort rates (that is, number of 
lines of code per budgeted effort unit and number of modules 
per budgeted effort unit) to determine whether enough effort 
has been allocated to complete the development. 

Managers can compute actual productivity rates to compare 
the pace of implementation with that experienced in past 
projects or with that budgeted. Productivity factors might 
include the number of lines of code in the projects' perma- 
nent source code libraries, the number of coded modules in 
the project libraries, or the number of completed pages of 
documentation per effort unit since the beginning of the 
implementation phase. 

® Growth rate of the number of lines of code . The 
growth rate of the number of lines of code in the project 
libraries is another indication of the pace of the project. 

o Error rate (number of errors per 1000 lines of 
code) . 
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Q Number o£ changes to code in the project’s perma- 
nent source code libraries . Managers can use the error rate 
and the number of changes made to code after it has been 
placed under configuration control as indications of the 
code's reliability and stability. Excessively high figures 
for these measures {in comparison to past projects) might be 
caused by inadequate design specifications or insufficient 
testing by developers. 

/ 

o Number of identified discrepancies versus number of 
resolved discrepancies . The number of discrepancies identi- 
fied in release testing is also a measure of the system's 
reliability. A widening gap between the number of discrepr 
ancies identified and the number of discrepancies resolved 
as implementation progresses probably indicates problems 
requiring the manager's attention. 

o Computer usage rate (number of minutes per 
1000 lines of code) . A computer usage rate much lower or 
much higher than previous projects may indicate problems in 
development, such as insufficient testing or excessive num- 
bers of diagnostic test runs. 

The SEL recommends the use of all these concrete measures. 
The SEL does not advocate the use of the more abstract meas- 
ures of the development product (for example, the McCaoe and 
Halstead measures) because a clear understanding of their 
meaning has not yet been obtained. 

Managers must monitor the progress of the development 
throughout the staged implementation process. The detailed 
chart maintained by the librarians as part of the unit de- 
velopment folaers, showing the exact status of each module 
in the system, contains the information necessary to assess 
how complete each ouild/release is. At all times throughout 
the implementation process, managers must knov/ where the 
project is (that is, its exact status) and where the project 
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is going (that is, the detailed schedule for completing the 
project) . ’ 

3. 4. 5. 2 Evaluation Criteria 

To evaluate the quality and completeness of the products of 
implementation, managers must consider the following ques- 
tions: 

I 

o For source code 

Is the code complete? 

Does the code adhere to the design? 

- Does the code adhere to the coding standards? 

How reliable is the code? What is the con- 
fidence level of the system performing without 
failure? 

- Is the code maintainable? How easily can 
changes be introduced, tested, and verified? 

How stable has the code been? 

o For test plans and results 

Are the test plans complete? Is all necessary 
information provided for each test? (See Sec- 
tion B.6 of Appendix B.) 

Are the tests specified in the test plans re- 
peatable? If two different groups execute the 
test plans, will the same tests be performed? 

- Do the test plans cover the key functional 
capabilities of the system? 

- Have the results of release tests been re- 
viewed by developers and managers for discrep- 
ancies? 
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o For documentation 

Does the documentation contain the key infor- 
mation? 

Is the documentation as brief as possible? 

Is the documentation clear and easy to under- 
stand? Can it be used by someone not familiar 
with the system? Tliat is, is each document 
styled for its intended audience? 

3.4.6 KEY MANAGEMENT ACTIVITIES 

Several key management considerations during the implementa- 
tion phase are identical to those in the preceding life 
cycle phases and are repeated below. Tlie activities include 
both planning and monitoring. Specifically, managers 

o Ensure adherence to 

Reporting procedures. 

- Data collection procedures. 

Quality assurance procedures. 

Coding standards. 

Configuration management procedures. These 
procedures--especially change control on the 
project's permanent source code libraries-- 
must be enforced during the implementation . 
phase when the staff is at its peak size and a 
large amount of code is being produced. 

o Monitor adherence to the planned schedule, monitor 
expenditure of resources, and update cost and resource es- 
timates and schedules. As implementation progresses, it 
becomes easier for managers to estimate the size of the sys- 
tem. Actual resources expended and progress during imple- 
mentation can also be obtained to update cost and resource 
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estimates with a resource estimation model (like the SEL 
Meta-Model). Updating cost and resource estimates, with 
resulting updates to schedules and staffing plans, is neces- 
sary several times in this phase as various builds and re- 
leases are completed. 

o Ensure that all facets of the project are com- 
pletely visible (that is, know exactly v/here the project is 
and where it is going at all times) . Project visibility is 
critically important. Managers must know at all times the 
exact status of all task activities and the detailed plans 
for development completion. This is necessary so that prob- 
lems can be dealt with when they occur rather than late in 
the process, when their impact is likely to be greater. 

New management activities specific to implementation include 
the following: 

o Review the release and system test plans and par- 
ticipate in the test result reviews for each build/release. 

j 

o Resolve discrepancies identified by the build/ 
release testing. 

o Plan the transition to the system testing phase . 
Managers must ensure that the data is available to perform 
the tests specified in the system test plan and that ar- 
rangements have been made to provide all computer resources 
required for system testing. They must inform development 
team personnel of the testing procedures to be followed and 
provide them with required training. Special emphasis is 
olaced on enforcing the strict change control procedures for 
the project's online source code libraries during final re- 
lease testing and system testing activities. 


3-55 


910R 





Er<!D pnoDUcre 


ACTSGE^JS AWD TRArJSACTSOCyS 

CV^cni Created 

Systotn Test risji EKecutsd 

Cbcrcponclss Elcsoivisd 

User's Guide Reviawod end Revised 

System Description Roviawsd end Revised 

Accoptonc® Testing PJonned 


System Cods Updata 

Supporting Data end System Fiiss Update 

System Tost P!en nosults 

User's Guido Updato 

System DossriptSon Updsto 

SofOtvaro DaveSapment Plan Update 



Project F\3otcboo!( 

Data CoSisctlon 
Ubrerisna 

Unit Dovoiopmsnt Foidors 

Hequirements Question end Changa Rocerds 

Design Decision end Chonge Rcccrds 

Cotlo Chango Records 

Ccnflguraticn r^&nagemont 

Formal Test Plan 

FunctioncI (Thread) Testing 


PDL Processor 

Sourco Code Librcry Mnnegemont System 

Structured Coding Language 

CAT 

Resource Estimation 



rilF-CEDINC. rAGK BLANK NOT FILMEE? 


3-57 













tVIAS^AGSaiSr^T 


ACTEOrJS AR3D Ti;?AaSSAGTI©El3S 


AC7iV!TtES Systesn Tost Kjot fSc-cute Rov!avs?od 

Discrepoitcio Rccasitfci! 

Acisplfaioo Tost Pten Rouiowcd 
AccaptsncQ TcssSng TrGnoielon PSsrmsd 

User's Guld® RQvFevjctI 
Gyatem DeaerSpUon RovFewesS 

TDD Etoma Rcso5wod 

Rc: 5 «!remonts Ch®R5$3 Rovfoafsd end ABS®esed 
Dsclgn Chances Ravtewed and IVcSkerJ Throtsgh 
Cods Chsngcs RovEswed 

Team Trninsd 

Stsndards end Preesduras Enforced 
Pros^eea Pilonltorod’ 

Vlalbilfty Promatod 
System Steo Estimatod 
Rosourccs end Cost Estimated 
Team Intarection Coordinated 

MEASURES TOO Itsms 

0ct;uircmsnt3 Chances 
Requiremonts Questions and Answoi-s 
Design Changes 
Coda Changes 
Test Compistion Checklist 
Cods GrovAh Roto 
Error/Chango Growth Rotes 
Discropcncios/Rcsclutions Growth Rates 
Computer Usage Growth Rate 
Team /Individual Productivity Rates 
Subjectivo Evaiuations 

MC3-(51>-83 


3-58 









System Testing 


The recommended softv;are development guidelines for the sys- 
tem integration and testing .phase are described in detail 
belov.'. 

3.5.1 MAJOR ACTIVITIES 

System integration and testino begins at the end of the im- 
plementation phase. At this point, all code for the system 
is complete, and the release test plan for the last system 
release has been executed satisfactorily by the 'developers. 

In this phase, the developers validate the completely inte- 
grated system by functional testing of the end-to-end system 
capabilities according to the system test plan prepared dur- 
ing the preceding life cycle phase. Specifically, the de- 
velopment team 

® Builds the system from the project's permanent 
source code libraries 

o Performs the tests specified by the system test plan 
o Reviews the test results 

I 

o Corrects code to fix any errors identified by the 
system tests 

o Revises the drafts of the user's guide and system 
description, if necessary, so that the documenta- 
tion reflects the final state of the system 

© Prepares for the acceptance testing phase 

System testing, which proceeds according to the system test 
plan, is performed like the testing of each release during 
the implementation phase. The development team and its man- 
agers, including customer and contractor personnel, care- 
fully review the test results to identify any discrepancies 
between the way the system works and the way it is supposed 
to work. The developers then correct the errors in the code 
that are causing these discrepancies. The system testing 
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phase is complete when all tests in the sys • test plan 
have been executed successfully. Since the system test plan 
must specify the expected output and the critecia for 
determining whether or not the test v;as successful (see Sec- 
tion B.6 of Appendix B) , the conditions for system integra- 
tion and testing completion are not ambiguous. 

Toward the end of this phase, the development team must pre- 
pare for the beginning of acceptance testing. They must 
become familiar with the acceptance test plan and the ac- 
ceptance test procedures. They must obtain the computer 
resources necessary for acceptance testing and modify the 
JCL, command procedures, and so on, to perform the accept- 
ance tests. The development team must also begin to in- 
struct the acceptance test team--by demonstrations and 
documentation--in the system's operation. 

3.5.2 END PRODUCTS 

At the end of the system testing phase, the completed system 
is available. The only new product of this phase is 

o Test results from the system test plan 

The remaining products ate updated- versions of products pro- 
duced during implementation: 

© Completed code for the. system, including changes 

made to correct discrepancies identified by system 
testing 

o Supporting files necessary for building and execut- 
ing the system (for example, JCL, command proce- 
dures, and load modules) 

© Updated drafts of the user's guide and system de- 
scription, reflecting the state of the system at 
the completion of system testing 
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3.5.3 METHODOLOGIES 

The methodologies used during system testing are a subset of 
those used during the implementation phase: 

© Project notebook. 

© Data collection. 

© Librarians. 

o Unit development folders. 

© Formal recording of changes. 

o Functional (thread) testing. 

o Configuration management procedures . Strict ad- 
herence is essential. Because all code is under configura- 
tion control at this time, any changes to the code in the 
permanent source code libraries must be made according to 
the established procedures and must be recorded by means of 
development change forms. The configuration control proce- 
dures used must ensure that the load modules being tested 
correspond to the code in the project's libraries. Although 
requirements and design changes are not frequent this late 
in the life cycle, when they do occur, the same formal re- 
cording mechanisms for requirements questions and changes 
and design questions and changes must be used as is recom- 
mended in the preceding life cycle phases. 

© Formal test plans . The system test plan is the 
basis for system testing. The tests specified are designed 
to verify the system's end-to-end functional processing ca- 
pabilities or threads. The system test plan is written and 
carried out by the developers. The system test plan fre- 
quently contains a number of tests specified in the build/ 
release test plans (see Section 3. 4. 3. 6, page 3-49). 
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3.5.4 TOOLS 

Managers continue to use the follov;ing tools: 

o An online configuration management tool (for ex- 
ample, CAT) 

\ 

o An online source code library management system, if 
available 

3.5.5 MEASURES 

The following subsections present various measures and eval- 
uation criteria for assessing system testing. 

3. 5. 5.1 Objective Measures 

The objective measures that managers must monitor during 
system testing are the same as those of the implementation 
phase (see Section 3.4.5. 1, page 3-51): 

o Actual productivity rates for the completed system 
versus planned productivity rates (number of lines of code, 
number of modules, and number of pages of documentation per 
effort unit) . 

o Error rate (number of errors per 1000 lines of code) 

® Number of changes to code in the project's perma- 

nent source code libraries. 

o Number of identified discrepancies versus number of 
resolved discrepancies. 

o Computer usage rate (number of minutes per ' 

1000 lines of code) . 

0 Actual size of completed system versus planned size 
(number of lines of code, number of modules, and number of 
pages of documentation) . Comparing actual versus planned 
system size and productivity rates enables managers to eval- 
uate the accuracy of the process they used to estimate sys- 
tc-'-' 'r-e, resources, cost, and schedules. This information 
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adds to existing historical knov/lcdge about the estimation 
process and can helji to make this process more accurate for 
future projects. The actual computer usage rate for the 
completed system can also be useful in estimating required 
computer resources for future projects. 

3. 5.5. 2 Evaluation Criteria 

Because the products of this phase are basically updated 
versions of those produced during implementation, the sub- 
jective criteria for evaluating their quality and complete- 
ness are similar to those used in the preceding life cycle 
phase (see Section 3. 4. 5. 2, page 3-53). Managers can con- 
sider the following questions; 

o How reliable is the code? What is the confidence 
level of the system performing without failure? 

o Is the code maintainable? How. easily can changes 
be introduced, tested, and verified? 

o How stable has the code been? 

Q Have the configuration management procedures for 

the system been strictly followed? Are the source 
code and load modules consistent? 

o Have the results of the system tests been thoroughly 
reviewed by developers and managers for discrep- 
ancies? 

o Do the test results meet the system requirements 
for each test in the system test plan? Does the 
system satisfy all requirements? 

o Is the documentation complete and correct? Does it 
reflect the state of the completed system? 
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3.5.6 KEY MAIIAGEMENT ACTIVITIES 

The manager's primary concerns during this phase are iden- 
tical to those in the preceding life cycle phase and include 
both planning and monitoring. Specifically, managers 

® Ensure adherence to 

Reporting procedures 
Data collection procedures 
Quality assurance procedures 
Guidelines/standards 

Configuration management procedures, espe- 
cially change control 

o Monitor adherence to the planned schedule and ex- 
penditure of resources, and update cost and resource esti- 
mates and schedules 

o Ensure that all facets of the project are com- 
pletely visible i 

© Review test results for each test in, the system 
test plan 

o Review system documentation 

o Resolve discrepancies identified by system testing 

o Plan the transition to the acceptance testing phase. 
Managers must ensure that the data is available to perform 
the tests specified in the acceptance test plan and that 
arrangements have been made to provide all computer resources 
required for acceptance testing. Transition planning is 
especially important for the acceptance testing phase be- 
cause the development team must work with two different 
groups 'the acceptance test team and maintenance and opera- 
tion personnel) . Managers must ensure that the procedures 
to be followed during acceptance testing are well defined 
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and understood by the development team. Managers must also 
supervise the instruction'" of the acceptance test team and 
operators in the system's operation. Providing this in- 
struction is the developers' responsibility. Special 
emphasis is placed on enforcing the strict change control 
procedures for the project's online source code library 
during system testing and acceptance testing activities. 
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The recommended software development guidelines for the ac- 
ceptance testing phase- are described in detail below. 

3.6.1 MAJOR ACTIVITIES 

Acceptance testing begins at the end of the system testing 
phase, v/hen all tests in the system test plan have been exe- 
cuted satisfactorily by the developers. Before acceptance 
testing begins, an acceptance test plan is prepared by the 
acceptance test team, based on the information in the func- 
tional specifications and requirements document. The system 
is then tested according to this plan. 

During acceptance testing, an independent acceptance test 
team tests the system to validate that the software meets 
all requirements. The development team assists the accept- 
ance test team. Specifically, the development team 

o Builds the system from the project's permanent 
source code libraries 

o Provides training for users and operators 

0 Sets up and executes tests as specified in the ac- 
ceptance test plan at the direction of the accept- 
ance test team 

« . Participates with the acceptance test team in. the 

reviev/ of the test results to identify discrepancies 

o Corrects the code to. fix any errors identified by 
the acceptance tests 

o Provides user assistance to the acceptance test team 

0 Completes the final versions of the user's guide 

and system description 

© Delivers the final system to the customer 


3-69 


9108 



Acceptance Testing -,, 

!-.i‘ I *, • V .'-.I 

Four important activities are described in more detail below. 

© Agree on tost procedures . Before acceptance test- 
ing begins, the procedures for acceptance testing must be 
agreed on by the managers of both the development and the 
acceptance test teams and given to the team members. The 
procedures must specify whether all tests will be run before 
code is changed to resolve discrepancies. If modifications 
to the code are allowed as testing progresses, the effect of 
these modifications on the testing process must be ad- 
dressed. (For example, will acceptance testing start over 
after each modification or will all tests be completed be- 
fore they are rerun?) The procedures must also specify the 
respective responsibilities of the development and the ac- 
ceptance testing team members and the lines of communication 
between these two teams, their managers, and the operations 
personnel with whom the teams must work to perform the ac- 
ceptance testing. 

o Understand the test plan . The acceptance test plan 
prepared by the acceptance test team is similar to the re- 
lease and system test plans prepared by the development team 
(see Section D.6 of Appendix B) . For each test to be per- 
formed, the acceptance test plan must specify the purpose of 
the test (that is, the specific functional capabilities or 
requirements being tested), detailed descriptions of the 
input and required environment, and the operational proce- 
dure to be used (also see Reference 20) . The acceptance 
test team supplies test data for the acceptance testing and 
provides the development team with all. required external 
data sets. The development team sets up and performs the 
tests according to the specifications in the acceptance test 
plan . 

The acceptance test plan must also specify the expected out- 
put for each test to be performed and the criteria for 
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determining whether or not the test results are acceptable. 
The development and acceptance test teams must be able to 
agree on which discrepancies identified by the testing must 
be corrected before the system is accepted. The acceptance 
testing phase is considered complete when all tests speci- 
fied by the acceptance test plan have executed successfully. 
Because the acceptance test plan contains pass/fail criteria 
for each test, the conditions for acceptance testing comple- 
tion are not ambiguous. 

e Deliver the system . After the successful comple- 
tion of acceptance testing, the developers formally deliver 
the accepted system to the customer. They clean up all 
files and they prepare and deliver a system delivery tape. 
They also deliver the final versions of the user's guide and 
system description (see Section 3.4.2, page 3-45). After 
the system has been formally delivered, it becomes the re- 
sponsibility of a maintenance and operation group. The 
maintenance and operation phase of the software development 
life cycle is not addressed in this document. 

o Participate in the operational readiness review 
(ORR) . After the successful completion of acceptance test- 
ing, an ORR is held to evaluate the readiness of the system 
to support operations. The ORR is attended by the develop- 
ment team managers, the acceptance test team managers, the 
maintenance and operation team managers, and others involved 
with the system. See Section A. 4 of Appendix A for details 
about the ORR. 
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3.6.2 END PRODUCTS 

At the end of the acceptance testing phase, the accepted 
system is delivered to the customer. Some products of this 
phase are updated versions of products previously begun: 

© Completed code for the accepted system, including 

changes made to correct discrepancies identified by 
acceptance testing 

® Supporting files necessary for building and execut- 
ing the system (for example, JCL, command proce- 
dures, and load modules) 

o Final version of the user's guide 

o Final version of the system description 

Three products are new: 

o Test results from the acceptance test plan, 
o System delivery tape . 

® Software development history . Within 3 months af- 
ter system delivery, the program manager, with input from 
the project manager, writes a software development history 
for the project. This report summarizes development and 
evaluates the technical and managerial aspects of the proj- 
ect from a software engineering point of view. The purpose 
of the report is to allow development managers to become 
familiar with successful and unsuccessful practices and to 
provide them with a basis for improving the development 
process and product. See Section B.9 of Appendix B for the 
format and contents of the software development history. 

3.6.3 METHODOLOGIES 

For acceptance testing, the SEL recommends the same method- 
ologies as for the system testing phase (Section 3.5.3, 
page 3-61) . 
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3.6.4 TOOLS 

The tools recommended' for use in the acceptance testing 
phase are the same as those recommended throughout the soft- 
ware life cycle; CAT and an online source code library man- 
agement system. 

3.6.5 MEASURES 

The objective measures and evaluation criteria recommended 
by the SEL for use during the acceptance testing phase are 
the same as those recommended for use during the system 
testing phase (Sections 3. 5. 5.1 and 3. 5. 5. 2, pages 3-62 and 
3-63) . 

3.6.6 KEY MANAGEMENT ACTIVITIES 

Monitoring is the key activity for managers during this 
phase. Several management activities are identical to those 
in the preceding life cycle phase. Specifically, the man- 
agers 

o Ensure adherence to 

Reporting procedures. 

- Data collection procedures. 

Quality assurance procedures. 

- Guidelines/standards. 

Configuration management procedures, espe- 
cially change control. The managers must en- 
sure that all changes are coordinated with 
acceptance testing activities and are made 
according to established acceptance testing 
procedures. 

o Monitor adherence to the planned schedule and ex- 
penditure of resources, and update cost and resource esti- 
mates and schedules. At the end of this phase, development 
is complete, and the actual cost and resource expenditures 

3-73 





Acceptance Testing 


throughout the project are known. Managers can compare 
these figures to the estimates produced during each life 
cycle phase to understand the estimation process better so 
that they can make more accurate predictions for future 
projects. 

o Ensure that all facets of the project are com- 
pletely visible. 

These activities are specific to the acceptance testing 
phase : 

® Participate in the review of -.(.‘ceptance test results 

o Resolve discrepancies identified by acceptance 
testing 

o Ensure cooperation among the various groups in- 
volved during this phase (for example, development team per- 
sonnel, acceptance test team personnel, maintenance and 
operation support personnel, and librarians) and their 
adherence to established acceptance testing procedures 

o Schedule and participate in the ORR , and ensure 
that all pertinent groups participate 
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Directing and controlling the execution of the development 
plan is the most important aspect of the development man- 
ager's job once a feasible plan is produced. By definition, 
requirements change. For some complex requirements, design- 
ing an .implementation may take several tries. In some 
cases, the design may not be implementable because of a 
change in computer hardware configuration or limitations. 
Certainly the development project staff changes, and the 
personality of individuals may change. In short, the devel- 
opment process is very dynamic. Therefore, the successful 
execution of even the most complete plans involves 

o Carefully measuring or assessing development prog- 
ress and team performance (see Section 4.1) 

o Recognizing the danger signals or warning signs of 
problems that will prevent proper execution of the 
plan (see Section 4.2) 

® Taking appropriate steps to solve problems once 
they have been accurately identified so that the 
real problems are addressed — not their symptoms 
(see Section 4.3) 

Data Collection 

If the development manager expects to manage a development 
project successfully, he or she must measure it in some con- 
crete way to assess general and specific progress. Measur- 
ing the performance of team members is also essential, since 
it will indicate team strengths and weaknesses, areas for 
training, and potential problem areas. To measure the 
process in a concrete manner, the managers muse collect ap- 
propriate data, monitor its collection, and then use the 
data (1) for comparison with past projects, (2) in predic- 
tive models, and/or (3) with conventional techniques cf 
progress assessment. 
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Management and Control 

The SEL stresses the importance of collecting and archiving 
data throughout the so£tv/are development process, not only 
for monitoring the development project, taut also for under- 
standing the environment and evaluating the effects of new 
technologies on productivity and reliability. Key data 
types to be collected throughout the project include 

o Basic project statistics (estimated and actual num- 
bers of modules and source lines of code and the 
start and end dates for each life cycle phase) 

e Resource data (weekly expenditures of resources for 
technical staff, managers, secretaries, librarians, 
publications personnel, and computer usage) 

o Change and growth data (the number of source lines 
of code and number of modules entered into the sys- 
tem by week and the number of changes made to the 
system by week) ' 

o Activity data (weeki.y summary of hours spent in 
various project activities by each member of the 
technical staff) I 

o Change data (change report forms for all changes to 
the design and code after it has been placed under 
configuration control) 

o Subjective evaluation data from managers after com- 
pletion of project phases 

The SEL' s Guide to Data Collection (Reference 21) provides 
further recommendations on this topic. 
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Status Indicators , , ,, , 

f • 

Tlio preoedin<.j page contains a brief list of measures that 
the SliL has found beneficial in monitoring the status of 
development projects. Continually lower frequencies of 
change, smaller fluctuations in estimates, and smaller 
discrepancies in planned-versus-actual reports (i.e., con- 
vergence toward goals) as development i regresses are strong 
indicators that tlie development team has the project under 
control. Opposite tendencies usually indicate the develop- 
ment of problems. 

The development status indicators are described below. 


Frequency of schedule/milestone changes . During the devel- 
opment life cycle, the development managers are able to make 
better estimates of system size and the effort required for 
development; therefore, they are able to make better esti- 
mates of completion dates. Altliough the estimates are ex- 
pected to change periodically, the frequency and magnitude 
of the changes siiould continually decrease throughout the 
development life cycle. Uy monitoring this simple data 
point closely, especially once implementation starts, a man- 
ager may be able to identify problems early. 

Consistency in organizational structure compared with 
original plans . The development managers usually organize 
their personnel before a project starts and make minor ad- 
justments wliile preparing the development plan during re- 
quirements analysis and preliminary design. Substantial 
changes to those plans usually iniiicato problems. One 
Cvimraon example is the unpl.innod appearance of a senior group 
of personne l--moro senior than the development team--whose 
advertised task is quality assurance, a difficult design 
problem, or an iiulopendent assessment. A second example is 
the presence of a senior persv^n who appears to be running 
the operation but who has no ofticial role. A third ex- 
ample IS diffusion of tJjo project manager's or leafier 's 
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responsibilities, i.e., delegation of nearly total responsi- 
bility for pieces of the system to other development team 
members. 

Fluctuation in project staff lovol and system size esti- 
mates . As work progresses throughout the development life 
cycle, the development managers are able to make better 
estimates of system size, required effort, cost, and 
schedule. The uncertainty in these estimates will decrease 
and the confidence in them will increase after each phase of 
development (see Figure C-1 on page C-8 of Appendix C) . 
Estimates that reach or exceed the normal limits of uncer- 
tainty indicate problems with plans, understanding of the 
project, or staff composition. 

History of number and type of TBD items for requirements and 
design . A large number of TBD items or a small number of 
severe TBD items for requirements usually indicates a system 
definition problem. During design, large numbers of TBD 
items or severe TBD items indicate a lack of understanding 
by, or inexperience of, the development team. 

Ease of access to information on project status, schedules, 
and plans . All development managers prepare a development 
plan and maintain a project notebook, v/hich are kept up to 
date in a central repository. Yet it is not uncommon to 
solicit information from a project manager or leader and 
receive the response "I'll have to check." The longer it 
takes the team leaders to respond, the more suspicious 
higher level managers should be about the quality and use- 
fulness of project plans and records. 

Frequency and amount of unusually long hours required or 
planned to attain certain objectives . The list of reasons 
for overtime hours is long and includes getting the most 
up-to-date material together for formal reviews, meeting 
major milestones, recovering from late software/interface 
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deliveries or hardware failures, and covering for staffing 
problems. Sometimes the overtime hours are necessary and 
expected; however, overuse of this practice is frequently 
indicative of problem's with the staff's qualifications, the 
staff level, or the team's leadership. (Inexperienced proj- 
ect managers/leaders, who are the most qualified to perform 
most development tasks, frequently do certain functions 
themselves because they think it is faster that way, rather 
than enlisting the team’s help.) 

Level of detail (both technical and managerial) understood 
and controlled by the project manager and project leader . 

All development managers prepare a development plan and 
maintain an up-to-date project notebook. The project man- 
ager's responsibilities are technical consultation and man- 
agement of the development plan (technical, management, and 
configuration control approaches). The project leader’s 
responsibilities are technical direction and day-tc-day 
supervision of project activities. Yet it is not uncommon 
for the team leaders to be unable to respond to queries at 
status meetings. The more frequently the team leaders are 
unable to respond, the more suspicious higher level managers 
should be about the level of control that the team leaders 
have over the project. 

Discrepancies in staff level and v/orkload . All development 
organizations use some algorithm for determining staff 
levels based on the type and amount of each type of work to 
be done. The workload and staff levels, which are recorded 
in the development plan, change throughout the development 
life cycle. Discrepancies between these and the plan 
indicate problems. 

Discrepancies in planned weekly staff level and computer 
usage or in comparison with past projects . A decrease in 
the weekly staff level may indicate a temporary or permanent 








Status Indicators 




loss of personnel from the project to another project; an 
increase may indicate an attempt to remedy a deficiency. A 
decrease or slow start in using the computer may indicate 
that the development team is engaged in some other activity 
(for example, design) rather than testing. 
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The SEL has monitored many softv;are development projects, 
some of which were considered very successful and some of 
which were considered .less than successful. From this ex- 
perience, the SEL has compiled a brief list of indicators 
(preceding page) that often characterize serious problems 
within the project. 

These indicators are described below. 

Scheduled capabilities delayed to a later build/release . 
Assuming that a build/release approach to implementation is 
being followed, the single most important signal of seriou^ 
problems during implementation is rescheduling capabilities 
from one build/release to a later one. Although it is some 
times necessary to reschedule capabilities, the consistent 
rescheduling of capabilities from one build/release to 
another as a completion date nears often indicates serious 
problems. 

Coding started too early . It is not uncommon for develop- 
ment projects to be fully staffed too early or overstaffed, 
although it seems as though most projects are understaffed 
or staffed too late. Starting coding too early, i.e., be- 
fore a design from which coding can start has been approved 
is a signal that the development team will end up building 
on the structure of a system that is not best suited to 
satisfy the total requirements. 

Numerous changes made to the initial software development 
plan . When the development managers make numerous changes 
to the initial development plan, it is a signal that they 
are inexperienced andare reacting to internal problems 
rather than solving them. 

Guildelines or planned procedures deemphasized or deleted . 
When the development managers suggest that the deletion or 
deemphasis of a i othod or procedure will save time and help 


Danger Signals 


to make a deadline, it is a nearly certain signal that the 
deadline will be made in a questionable manner, if at all, 
and that makeup work will be needed later to complete the 
activity correctly. ^ ^ 

Sudden changes in staffing (magnitude) suggested or made . A 
sure signal of serious problems is the sudden suggestion or 
application of unplanned staff increases. 

Excessive documentation and paperwork with little direct 
bearing on required documentation prepared . When develop- 
ment managers suggest the complete, detailed, formal docu- 
mentation of each activity, it is a signal that they are 
inexperienced and that cost and schedule problems are immi- 
nent. Complete, detailed, formal documentation does not 
ensure success when the team leaders' effort is diverted 
from managing the technical aspects of the project. 

Continual increase in numbers of TBD items and ECRs meas- 
ured . A continual increase in the number of TBD items is a 
clear signal that technical problems are not being re- 
solved. A continual increase in the number of TBD require- 
ments and engineering change requests (ECRs) is a signal 
that the requirements are not adequately defined or stated. 

Decrease in estimated effort for system testing suggested or 
made . When the development managers suggest or make a de- 
crease in the estimated effort for system testing, it is a 
signal that they are inexperienced or they are excessively 
scheduling to success. With acceptance testing, system 
testing is the most sequential phase in the development 
process. Little can be done to compress the full testing 
phases. Assuming that testing can be compressed to make up 
for slippages in earlier phases leads to a loss of credi- 
bility in the development organization when the system is 
not ready on time or is flawed in operation. 
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Reliance on other sources for soon-bo-be-available soft- 
ware . Every experienced developer and manager recognizes 
the cost benefits of using existing software or soon-to-be- 
existing software. Kov/ever# all management levels of the 
development team must be especially concerned when the 
successful execution of their development plans depends on 
other sources for their system's software capability. 
Managers' concern should increase inversely proportionately 
to the level of control they have over the source who is 
developing the capability. For example, a loosely rej.ated 
project in the same organization is greater cause for con- 
cern than a closely related project in the same development 
organization; another contractor or a vendor is an even 
greater concern. The obvious problems with externally de- 
veloped software are (1) it is always late and (2) it is 
never fully checked out. 
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Corrective Measures 


Once the development manager has recognized that there is a 
problem, he or she must correct the problem. The preceding 
page contains a brief list of corrective measures that tne 
SEL has found effective. Depending on the problem, one or 
more of these corrective measures may be necess-iry. 

Frequently, when development managers find their projects in 
difficulty, they have a tendency to 

o Shortcut procedures, i.e., to cut out the presumed 
"busy” or nonessential work and to concentrate on 
the "real" work 

o Add staff (usually junior level) to help bail out 

© Plunge ahead to meet milestones with some kind of 
product 

The SEL's experience, however, shov/s that, more often than 
not, these steps compound problems. The corrective measures 
(preceding page) suggested by the SEL are usually counter to 
the normal tendency; that is, they increase and tighten pro- 
cedures, reduce staff levels (or replace junior with senior 
staff) , and slow down the process to get a better handle cn 
it or to better define the obj‘=‘Ctive. The following suosec- 
tions list the basic development problem areas and the sug- 
gested steps to correct them. 

4.3.1 BASIC PROBLEM AREAS 

The following is a list of the basic problem areas: 

1. Development plan problems 

2. Requirements or design problems 

3. Confusion with 

a. Development plan 

b. Requirements or design 

c. Development plan execution 
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I 

' i 

4. System growth problems because of I 

i 

a. Poor direction 
' b. Staff ability 

c. Major requirements changes 

d. Many minor requirements changes 

e. IncompTete facets of project 

1 

5. Changes or decrease in scope of plans 

6. Configuration problems 

7. Schedule problems 

4.3.2 STEPS FOR CORRECTIVE ACTION 

Following the outline above (Section 4.3.1), this subsection 
presents the steps to follow to correct problems in each of 
the basic problem areas. 

1. When there are serious problems with the developmenc 
plan , 

a. Stop development activity. 

b. Complete and/or review plans. 

c. Follow through with plans. 

2. When there are serious problems with requirements or 

design , 

a. Stop staff growth. 

b. Decide v/hich are appropriate: 

(1) Decrease the scope of the system. 

(2) Solve problems before proceeding. 

(3) Replace junior personnel with senior personnel. 

3 . When there is confusion , 

a. Obtain an accurate assessment of the cause. 
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3 . 


4 . 


When there is confusion (continued) , 

b. When the development plan is the cause of confusion , 

(1) Stop development activity. 

(2) Complete and/or review plans. 

(3) Follow through with plans. 

c. t^hcn requirements or design is the cause of con- 
fusion , 

(1) Stop staff growth. 

(2) Decide which are appropriate: 

(a) Decrease the scope of the system. 

(b) Solve problems before proceeding. 

(c) Replace junior personnel with senior per- 
sonnel. 

d. When plan esiecution is the cause of confusion , 

(1) Decide which are appropriate: 

(a) Decrease staff size to a manageable level. 

(b) Replace junior team leaders with senior 
leaders . 

(c) Replace junior team members with senior 
people. 

(2) Create intermediate products and milestones 
for review. 

(3) Increase status reviews to improve direction. 

(4) Follow through with plans. 

When there is inadequate system growth (progress) , 

a. Obtain an accurate independent assessment (audit) 
of the problem. 
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4. When there is inadequate system growth (continued ) , 

b. When poor direction inhibits system grovjth ^ 

(1) Decide which is appropriate: 

(a) Decrease staff size to a manageable level. 

(b) Replace junior team leaders with senior 
leaders. 

(2) Create intermediate products and milestones 
for review. 

(3) Increase status reviews to improve direction 
and to tighten management procedures. 

' (4) Follow through with plans. 

c. When staff ability inhibits system growth / 

(1) During design, replace junior team members 
with senior personnel to complete design. 

(2) During implementation, add intermediate- to 
senior-level personnel to complete implementa- 
tion. 

I 

(3) During testing, add senior personnel to solve 
problems and to improve direction. 

d . When major requirements changes inhibit system 
growth , 

(1) Stop staff growth. 

j (2) Decide which are appropriate: 

(a) Decrease the scope of the system. 

\ (b) Solve problems before proceeding. 

(c) Replace junior personnel with senior per- 
sonnel. 
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4. When there is inadequate system growth (continued), 

e* When many mine- requirements changes inhibit system 
growth , 

(1) During design and early implementation, 

(a) Stop staff growth. 

(b) Decide which are appropriate: 

i. Decrease the scope of the system. 

ii. Solve problems before proceeding. 

iii. Replace junior personnel with 
senior personnel. 

(2) During implementation, hold changes and com- 
plete implementation of a build of the system 
first. 

(3) During testing, hold changes and complete 
testing of a version of the system first. 

f . When ah incomplete facet inhibits system growth , 

(1) Decide which is appropriate: 

(a) Redirect senior personnel from less 
important or low-priority work to com- 
plete design or implementation. 

(b) Add senior personnel to complete design 
or implementation. 

(2) During testing, add senior personnel to solve 
problems. 

5 . When there is a significant change or decrease in the 
scope of the development plan , 

a. Obtain an accurate assessment of motivation. 
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5. When there is a significant change or decrease in the 
scope of the development plan (continued) , 

b. When confusion causes change of plan , 

(1) When the development plan is the cause of con- 
fusion , 

(a) Stop development activity. 

(b) Complete and/or reviev; plans. 

(c) Follow through with plans. 

(2) When requirements or design is the cause of 
confusion/ 


(a) 

Stop 

staff growth. 

(b) 

Decide which are appropriate: 


i. 

Decrease the scope of the system 


ii. 

Solve problems before proceeding 


iii. 

Replace junior personnel with 


senior personnel. 

(3) When development plan execution is the cause 
of confusion 

(a) Decide which are appropriate: 

i. Decrease staff size to a manageable 
level . 

ii. Replace junior team leaders with 
senior leaders. 

c. VJhen inadequate system growth (progress) causes 
change of plan , 

(1) Obtain an accurate independent assessment 
(audit) of the problem. 

(2) When poor direction inhibits system growth , 
follow steps (1) through (4) in item 4b. 
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c. When inadequate system growth (progress) causes 
change of plan;(continued) , 

(3) VJhen staff ability inhibits system growth , 
follow steps indicated in item 4c, 

(4) When major requirements changes inhibit system 
growth , follow steps (1) and (2) in item 4d. 

(5) When many minor requirements changes inhibit 

system growth , follow steps indicated in 
item 4e. * 

(6) When an incomplete facet inhibits system 
growth , follow steps indicated in item 4f, 

6. When there are problems with configuration control , 

a. Obtain an accurate assessment of weak areas. 

b. Firm up and tighten configuration management proce- 
dures. 

c. Follow through with plans. ' j 

7. When there are problems in maintaining schedules , 

a. Obtain an accurate independent assessment (audit) 
of cause. 

b. When confusion causes schedule slippage , 

(1) When the development plan is the cause of con- 
fusion , 

(a) Stop development activity. 

(b) Complete and/cr review plans. 

(c) Follow through with plans. 
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' ' ) 

b. When confusion causes schedule slippage (continued ) , 

(2) When requirements or design is the cause of 
confusion , 

(a) Stop staff growth. 

(b) Decide v;hich are appropriate: 

i. Decrease the scope of the system. 

ii. Solve problems before proceeding. 

iii. Replace junior personnel with 
senior personnel. 

(3) When development plan execution is the cause 
of confusion , decide which are appropriate: 

(a) Decrease staff sire to a manageable level. 

(b) Replace junior team leaders with senior 
leaders. 

c. When inadequate system growth (progress) causes 
schedule slippage , 

(1) Obtain an accurate independent assessment 
(audit) of the problem. 

(2) When poor direction inhibits system growth , 
follow steps (1) through (4) in item 4b. 

(3) When staff ability inhibits system growth , 
follow steps indicated in item 4c. 

(4) When major requirements changes inhibit system 
growth , follow steps (1) and (2) in item 4d. 

(5) When many minor requirements changes inhibit 
system growth , follow steps indicated in 
item 4e. 

(6) When an incomplete facet inhibits system 
growth , follow steps indicated in item 4f. 
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The preceding sections of this document present the software 
development and management practices, techniques, and aids 
that the SEL has found beneficial. This section identifies 
key aspects of successful software development projects and 
discusses the application of the recommended approach. 

The following subsections contain three lists identifying 
key aspects of software development: 

o Ten key "Do's" for project success 

0 Ten key "Don'ts" for project success 

o Ten key points for assessing the quality of a 

project 

These lists are derived from SEL experience using the recom- 
mended approach. 

Section 5.1 lists and describes the 10 most important guide- 
lines for managing a successful development project? Sec- 
tion 5.2, the 10 most important things to avoid in managing 
a development project. Section 5.3 highlights the 10 key 
points most useful in evaluating or assessing (auditing) a 
software development project. No particular order of prior- 
ity is implied in any of these lists. 

Section 5.4 discusses the application of the recommended 
approach to software development. 
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10 DOS 


The SEL's 10 most important "DOs" for project success are 
described belov/. ; ■ ' 

Use a small senior staff for the early life cycle phases . A 
small group of experienced senior personnel is better 
equipped to determine the approach, to prepare the software 
development plan, to sot priorities and organize the work, 
and to establish reasonable schedules. With a large team, 
there is a tendency to begin design or coding to keep people 
busy before the actual problem is known. 

Develop and adhere to a software development plan . This 
plan defines project organization and responsibilities; life 
cycle phases, approaches, intermediate and end products; 
approach guidelines and standards; product completion and 
acceptance criteria; configuration management procedures; 
mechanisms for accounting status; product and progress re- 
views; cost and schedule reviews; and . contingency plans. 

All development team members must know the plan and adhere 
to it. 

Define specific intermediate and end products . Specific 
intermediate and end products for each life cycle phase give 
the deve' apment team well-focused short-term goals, provide 
the team with a sense of accomplishment, and provide a means 
to measure and evaluate progress. 

Examine alternative approaches . Alternative approaches, and 
the rationale for them, must be considered and evaluated in 
terms of project objectives and constraints, such as 
schedule, cost, team skill mix, availability of resources, 
and existing software. This is especially important during 
design. Do not assume that there is only one way of per- 
forming the task; seriously examine at least one other ap- 
proach to the design. 
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Uso formal testing . Because all testing (unit, system inte- 
gration, acceptance) makes up 40 to 60 percent of a completed 
project's effort, cost, and schedule, it must be a well- 
organized and efficient process. Avoid a haphazard approach 
to testing; develop a test plan and follow it. 

Uso a central repository . Keep all development records and 
materials available in a central location so that the devel- 
opment process and progress are visible to management. Keep 
the repository organized and up to date throughout the proj- 
ect. 

Keep a detailed list of TBD items . Classify TBD items by 
severity of impact in terms of system size, required effort, 
cost, and schedule and set priorities for their resolution. 
Assign appropriate personnel to resolve TBD items and follow 
their progress closely to ensure timely resolution. 

Update system size, required effort, cost, and schedule 
estimates . Do not insist on maintaining original estimates 
of the system size, required effort, cost, and schedule. 
Requirements do change, the composition of the development 
team changes, and problems are encountered throughout the 
project. Most important, more information is learned about 
the size and complexity of the problem as the project pro- 
gresses. Each phase of the life cycle provides new and re- 
fined information to improve the estimates and to plan the 
project more effectively. 

Allocate 30 percent of effort for integration and testing . 

The activities in the system integration and testing and the 
acceptance testing phases are the most sequential in the 
development process. These phases account for 20 to 
40 percent of a completed project's total effort, cost, and 
schedule; and little can be done to compress or reduce the 
work required in these phases. The code must be complete 
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and unit tested before entering the system integration and 
testing phase. Avoid- the common error of assuming that the 
integration and testing effort can be compressed to make up 
for slippages in the schedule during design and implementa- 
tion. 

Experiment , in an age of increasingly scarce resources, 
reviev/ effectiveness, identify areas for improvement, and 
take steps to make the improvements. Acquire new skills, 
examine alternative approaches, and test and evaluate 
changes. Try new techniques. Using the same methods this 
year that were used 2 or 3 years ago indicates a lack of 
growth. 


5-6 


9108 



© Ocm't O^erstcsff 


o Don't Ailovy Team Ellsmbsrs To Proceed in an 
yndisospiinod r«?1annsr 

o Don't BGisgate TGcSmicai Dstolis to Team li^omboFS 

© Don't Asswmo TSiat a Rigid Set of Standards Ensajms 
Sascocss 

o Don't Assosrao Tfjot a Large Amoiant of Oocomienta- 
tion Ensisres Socceos 

O Don't Dsuicto from tb© Approved Design 

o Don't Assume That Reiaiiing Standards Reduces 
Costs 

Q Don't Assume Tiiat the Pace \TJiii Sncroase Later in 
tho Project 

o Don't Assume Tliat SntermsdSate Scheiuie Siippaga 
Can Be Absorbed in a Later Phase 

O Don't Assume That Everything Will Fit Together 
Smoothly at the End 


3103-(51l-03 


5-7 





10 Don'ts 


The snr.'r, 10 mont imporfcant "Don’ta" tor project succesn are 
doficribod below. • 

Don't overataCt (onpooLally dnntjorouo early in dovelop- 
niont) . A s'.mall ijroiip of s?onior poruonnel is bettor equipped 
than a larqe ataff to orqanlzo and determine the direction 
of a project. When a large staff is assigned at the begin- 
ning of the project, tlie staff members usually begin design- 
ing some aspect o» the system before the actual problem is 
known. After a significant amount of the budget is spent, 
managers frequently are reluctant to admit that a mistake 
was made and that the work performed is unusable. Because 
of tills unwi 1 1 Ingness to discard the work and start over, 
the remainder of tlie project will be based on an invalid 
design tliat causes further problems throughout the project. 

D on't a ll ow team members to proceed in an undisciplined 
m anner . Developing very reliable high-quality software at 
low coi'.t is not a creative art. Katlier, it is a very dis- 
ciplined application of a set of refineil principles, 
methoils, practices, and techniques. Apply them. 

Don't do l< njate lechn i tr a I details to tea m membe r_s_ . !•' i r s t - 
lino managers must know the technical details of the proj- 
ect. Do not vie legate this aspect of tlie project to the 
member:; ol tlie «ievelopment team, especially to those on a 
junior l«'vel. 

D on't, as :; ume tlia t a rig i t! set of standards em a ires iuiccess . 
Uucco:;:; is not guaranteed by any development mi'thodology , 
practice, vir techniviue. These stamiards prom>de viisciplino 
aiul consistency in I lie process ami facilitate design v^;a^k- 
through.s, cod<’ reading, ami test evaluation. However, the 
expi.'f ienceil judgment:;; ami viet: i:; ions ol the project manager, 
the lU've lopnn'iit team li'ader, and otlnn si'iiior technical per- 
sonnt'l are necess.ary to en:;ur«i .■;tu"ci'ss ol the project. 
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10 Don'ts 


Don't assume that a large amount of documentation ensures 
success . Each phase of the life cycle does not necessarily 
require a formally produced document to provide a clear 
starting point for the next phase. The level of formality 
and amount cf detail to be provided in the documentation 
must be determined by the project size, the life cycle dura- 
tion, and the lifetime of the system. For example, 
intermediate-sized projects (4 to 12 staff-years of effort) 
of 18 months' duration or less do not require a formally 
produced preliminary design document. By the time the 
material is prepared (edited, typed, reviewed, and so on) , 
the design document is obsolete. 

Don't deviate from the approved design . As development pro- 
gresses, developers may tend to implement a slightly dif- 
ferent design that still satisfies the requirements. The 
managers must control this tendency by holding design walk- 
throughs. Modifications by individual developers may be 
correct in the local sense but not for the system as a whole. 

Don't assume that relaxing standards reduces cost . When a 
failure to meet a deadline seems imminent, managers and de- 
velopers sometimes attempt shortcuts by relaxing configura- 
tion control procedures, data collection procedures, design 
formalism, or coding standards. In the long run, panic ac- 
tions cause greater problems and added expense and do not 
usually succeed in making the deadline anyway. 

Don't assume that the pace will increase later in the proj- 
ect . When design, implementation, or testing is progress- 
ing slowly, assign additional senior personnel to help 
and/or make schedule adjustments. The workrate for a given 
activity is characteristic of the particular development 
team--it generally does not change within a short period of 
time. Do not assume that the team will work faster later on, 
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10 Don'ts 


Don't assume that infaecmediate gchedule slippage can be 
absorbed in a later phase . When some part of the design 
must be completed during- implementation, or when some part 
of the implementation must be completed during system test- 
ing, the later phase will not be completed on time unless 
extra staff is added well before its scheduled completion. 

It is a common mistake of managers and overly optimistic 
developers to assume that the team will be more productive 
later on. The workrate of the team cannot be changed ap- 
preciably because the project is approaching completion of a 
phase, especially in the later phases of development, when 
the process is most sequential. Because little can be done 
to compress the schedule during the later life cycle phases, 
the managers must change the schedule or apply additional 
staff as soon as the problem is known. 

Don't assume that everything will fit together smoothly at 
the end . Managers erroneously assume that late pieces of 
design, code, or testing will contain few or no errors and 
will fit into the system with minimal integration effort. 

The work of the developers will not be of higher quality 
later in the project than it was earlier. 
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Project Quality ; . 


The SEL's experience in assessing the quality of an active 
project results from close review of monitored SEL projects 
and from conducting audits or independent evaluations of 
other projects. Ten key items for assessing the quality of 
a project are explained belovi?. 

Software development plan . Is there a written software de- 
velopment plan? Do all team members know it, and are they 
adhering to it? 

Life cycle phases and products . Have the project managers 
and team leaders defined a life cycle with specific inter- 
mediate and end products? Are there centrally located lists 
detailing what these phases and products are? 

Managers and leaders . Is there someone in charge? Does the 
development team leader know 90 percent of the technical 
details; the status of all major pieces of the software; the 
status of critical, major, and nominal problem areas; and 
future needs and potential problems? Doss the project man- 
ager know the status of all major pieces of the software; 
two alternatives being considered to solve critical and 
major problems; one alternative being considered to solve 
nominal problems; the impact of critical and major TBD 
items; and the likelihood of cost and schedule perturbations 
in terms of workrate and workload? 

Staff sice . Are the correct number of people working on the 
project based on the projected workload and the development 
team's workrate? Are staffing changes planned to match pro- 
jected increases or decreases in workload? Do the project 
workload and staff workrate projections match the schedule 
projections? 

Project objectives and status . Do development team members 
|<now how their individual work fits in the project? Do they 



Project Quality 

know their own deadlines and the objectives of those dead- 
lines? Do they knov; when to expect data or interfaces to be 
established? Do the senior members of the team know the 
overall objectives of the project and hov/ it fits in with 
the work being done by other groups? Do they know the prob- 
ability of timely interfaces and the impact and contingency 
plans if they ace not established? 

Configuration control . Is there a written configuration 
control plan? Do all development team members know and 
follow it? 

List of TBD items . Is there a single complete list of TBD 
items? Are they classified by severity of impact in terms 
of system size, required effort, cost, and schedule? Who 
sets the priorities for the resolution of TBD items, and how 
is their resolution scheduled and tracked? 

Adherence to methodology . Do the development team members 
follow a specific methodology? Do all team members have the 
satae understanding of what the methodology is? 

Alternative approaches . Has the development team seriously 
considered at least one other design? Have they written 
down and justified the rationale for selecting the current 
design over alternatives? 

Contingency plans . Are there written contingency plans on 
how to continue project work if, for example, a severe or 
major TPD item breaks a development sequence, an interface 
is several weeks or months late, the computer is down or 
malfunctioning for an extended period of time, the configu- 
ration of the software system is lost, or a key team member 
leaves the team prematurely? Are the manager and team 
leader aware of a potential problem and do they have a plan 
to minimize its effect? 
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Recommended Approach 


The SEL does not expect users to transfer the SEL's recom- 
mended approach to software development and apply it to 
' their own environment without modification. The recommended 
approach is standard in the SEL environment for one class of 
software — flight dynamics systems. Even in the SEL environ- 
ment, however, the recommended approach is tuned to the 
characteristics of individual projects and modified to re- 
flect successful new technologies. 
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This appendix summarizes suggested formats and contents of 
reviews scheduled during the software life cycle. All the 
reviews are part of the recommended approach to software 
development. For each review — SRR, PDR, CDR, and ORR — the 
following areas are addressed; 

e Review presentation material 

o Review format 

e Review hardcopy material 

where the review presentation material is a subset of the 
review hardcopy material. Following each hardcopy material 
summary is a brief description of the contents. 









SCIIRI PeiESEOTATiOrd m/AY 


© Srsts-odifciijcn end Agenda 
O KoijuirGmonta SitiT!:nQry 
O AnsSysss Ovorviow 
© FuncticnaS Specmcadons 

— Snvironnrscntal Gonsidsraticna 
— OpsrationaS Ktjqujromsnta 

(1) Operating Scenarios 

(2) Data Flow AncSycaa 

(3) Pcrfosmanco RoquiremonSa 

(4} intsrfcco Rstiuiromants 

— Requirements Relationships 
o Derived System Requirements . 

© Requirements El^anasoment F2on 
o PorsonneS Organisatiors and Interfaces 
o Software Performance and Testing Roquiroments 
© Issues, TBD Items, and Prcbloms 
O iViiiestones and Suggested Oovelopment Schedulo 


SRR 

PRESOPJTERS 

Requirements Definition Team 

PARTICIPAWTS 

Dovolopmant Team Rsprocentotives 
Quality Assurance Representatives 
User Representatives 
Customer Roprcsenlativos 

TIR/5E 

After Functional Specifications Completed and 
Before Functional Design Started 

HARDCOPY 

DISTRIBUTIOPJ 

rViinimum of 5 Days Before SRR 
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1. Intircduciiosi 

2 . KGqisIron^cma Susvimarv 

3. Arssivcls Ovorviavv 
FuncticnsI SpC'CSfisatScns 

ta. Envircsrsmcnta! ConsJdofatfoias 

b. 0pcrat!ons8 KG-qujrosncnts 

(1) Opcsratlrsg Scsnarcoa 

(2) OctQ Row Acialycia 

(qS Syotcfn Bnpisl 

(b) Procoscfeg ElcqsisrcKtsnts 

(c) System Output 

(3) Porformanco Eicqulrcmonls 

(4) Intorfffico Roquli-omonto 

c. PScqujrsmcnta RolatlsnshSps 

5. Porfvcd System Requirements 

6. UtlUty, Support, end Test Programs ' 

7. iloitsablo Software Summary ’ ; 

8. Bate Sot Oefinitiens 
S. Requirements r<.^artagsment Plan 

a. Porsonno! Assignments 

b. Description of Required Dccumonts 

c. Configuration ControS Approach 

d. Enhcncoment/Pciasntononco Procedures 
o. Hoporting and Testing Evcluoticn Procedures 

10. Porsonno! Organization and Interfaces 

11. Softwaro Porfornianco and Testing Requirements 

a. Anctyticai 

b. System 

c. interface 

d. Accoptanco 

12. Issues, TBD Items, and Problems 

13. R/iilostonos and Suggested Development Schedule 
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SRR 


The SRR hardcopy material contains 

1. Introduction — Purpose of system, background of project, 
and outline of review material 

2. Requirements summary — Review of top-level (basic) re- 
quirements that are developed to form the functional 

; specifications 

a. Background of requirements--Overview of project 
characteristics, major events, support 

b. Derivation of requirements--Diagram showing 

(1) Project Office input used to formulate the 
requirements for the support organization-- 
support instrumentation requirements document 
(SIRD) , memorandums of information (MOls) , 
memorandums of understanding (MOUs) , etc. 

(2) Support organization input used to formulate 
requirements for the system engineering orga- 
nization, e.g., SIRD, MOIs, MOUs, and support 
organization's constraints, assumptions, and 
guidelines 

(3) System engineering organization input used to 
formulate the requirements (functional speci- 
fications and requirements document (FSRD) ) 
for the software engineering organization-- 
Analytical studies, software system analysis, 
etc . 

c. Type of requirements 

(1) Evolution of support requirements 

(a) Typical support 

(b) Critical support 
(C; Special support 

(d) Contingency support 
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(2) Operational support scenarios 

(3) Relationship of requirements matr ix--Relation- 
ship of top-level requirements to operational 
support scenarios 

d. Constraints, assumptions, and guidelines 

(1) Organizational interf aces--Organizations that 
provide system and support input and receive 
system output 

(2) Data availability for the operational support 
scenarios — Frequency, volume, format 

(3) Facilities — Target computing hardware, envi- 

ronment characteristics, communications proto- 
cols, etc. I 

(4) General software considerationr.--Kigh-level 
description of computer storage, graphics, and 
failure/recovery requirements; operator inter- 
action requirements; system error recovery and 
diagnostic output requirements; etc. 

(5) Support and test software considerations — High- 
level description of requirements for data 
simulators, test programs, support utilities 

e. Overview of functional specifications and require- 
ments document (FSRD) 

(1) History of evolution--Draf t dates and reviews 

(2) Outline of cont.rots 

3. Analysis overview--Matheraatical and logical framework 
necessary for design, implementation, and testing of the 
system 

a. Introduction--Project overview, support firsts, 

bases for analysis 
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b. Analysis appfoach”-Major areas of analysis neces- 
sary to produce FSRD 

c. Special studies and results--Overview of purpose of 
studies and conclusions 

4. Functional specifications 

a. Environmental considerations--Target computing 
hardware, special computing capabilities (e.g., 
graphics), operating system limitations, computer 
facility operating procedures and policies, support 
software limitations, data base constraints, re- 
source limitations, etc. 

b. Operational requirements 

(1) Operational scenar ios--High-level diagrams of 
operational support goals and concepts, in- 
cluding all interfaces 

(2) Data flow analysis--Diagrams showing input, 
processes, and output, including all interfaces 

(a) System input--Data availability, fre- 
quency, volume, coordinates, units, for- 
mats 

(b) Processing requirements--What functions 
must be performed to transform some input 
into some output 

(c) System output--Data frequency, volume, 
coordinates, units, formats 

(3) Performance requirements--System processing 
speed, system response time, system failure 
recovery time, output data availability 

(4) Interface requ irements--Summar y of human, 
special-purpose hardware, and automated system 
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intfirfacec, including references to interface 
agreement documents (IADs) and interface con- 
trol documents (ICDs) 

, c. Relationship of requirements matrix, e.g., rela- 
tionship of requirements to operational scenarios 

5. Derived system requireraents--Structured, enumerated list 
of the requirements derived in formulating the func- 
tional specifications 

6. Utility, support, and test programs--Rationale for par- 
titioning system into smaller programs to support opera- 
tional scenarios; to support data processing volume, 
frequency, or special conditions; and to make use of 
reliable existing software; also, rationale for data 
simulators and test programs 

7. Reusable software summary — Identification of existing 
software components that satisfy specific system func- 
tional specifications exactly or that will satisfy them 
after specified modifications 

8. Data set definitions for 

a. Interfaces external to the system, including 

(1) Format and description of items in header, 
data, and other records 

(2) File structure (blocking and access methods) 

(3) Storage requirements in all processing modes 

b. Interfaces between specified utilities and support 
programs--Format of data, description of data flow, 
etc. 

9. Requirements management plan 

a. Personnel assignments — Requirements definition and 
analysis team organization; key personnel and their 
responsibilities and start dates; etc. 
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b. Description of required documents — Name# form, and 
contents of documents; production standards; date 
scheduled for delivery; name of organization (per- 
son) responsible; internal, external, and quality 
assurance reviev; procedures; authorization proce- 
dure for release; etc. 

(1) System functional specifications and require- 
ments 

(2) Software system 

(3) Operational support 

c. Specifications/requirements change control proce- 
dures — Initiation, forms, reviews, approval, au- 
thorization, distribution 

d. System enhancement/maintenance procedures--Initia- 
tion, forms, reviev/s, approval, authorization 

e. Reporting and testing evaluation procedures — 
Forms, reviews, approval, authorization, distribu- 
tion 

10. Personnel organization and interfaces — Diagram showing 
organizational interfaces, their points of contact, and 
their responsibilities 

11. System performance and testing requirements--Organiza- 
tion responsible; testing philosophy and procedures; 
forms; internal, external, and quality assurance re- 
views; approval 

a. Analytical tests 

b. System tests 

c. Interface tests 

d. Acceptance tests 
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12. Issues, TBD items, and problems — A characterization of 
all those things that affect plans for preliminary de- 
sign and the state of the requirements, an assessment 
of their effect on progress, and a course of action to 
resolve them, including required effort, schedule, and 
cost 

13. Milestones and suggested development schedule--Reviews; 
delivery of interfaces, documents, and externally de- 
veloped software; data flows for readiness preparation 
and training 
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14. Personnel Organization and Interfaces 

15. Testing Strategy 

a. Goncrai Approach 

b. Ejxtcnt 

c. Contra! Sf^ochmiiems 

1G. Issues, TBD Items, and Problems 
17. iVlilostonos and Schedules 
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The PDR hardcopy material contains 

1. Introduction--Purpose of the system and outline of re- 
view material 

a. Requirements Gummary--Or igin and format of re- 
quirements; list of major system components, in- 
cluding the top-level (basic) requirements which 
they satisfy and which are covered in the review 

b. Additional derived software requirements--Require- 
ments derived by the development team during the 
requirements analysis phase 

(1) Operating scenario requirements--Data han- 
dling, execution frequency, turnaround time, 
checkpoint/restar t capabilities, graphics 
needs, etc, 

(2) Environment considerations — Target computing 
machine, operating , system, computer system 
support software, graphics packages and hard- 
ware, etc. 

(3) Software legacy (past experiences and history) 

(a) Cost f actors--Exper ience history, design 
models, reusable code, etc. 

(b) Schedule factors--Deadlines ; hardware, 
software, and support dependencies; etc. 

2. Design overview 

a. Requirements summary — List cross-referencing top- 
level (basic) requirements to major system compo- 
nents presented at SRR 

b. Performance requirements--Cross-reference list of 
performance requirements that led to partitioning 
of system into major components 
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c. Design dr ivGrs--Primary factors that influenced 
the development team's design, e.g., operating 
scenarios, environmental considerations, and soft- 
ware legacy 

3. High-level diagrams of operating scenarios--Input stim- 
ulus, processing, output stimulus, and interfaces to 
show how requirements are met 

4. High-level diagrams of system structure — Internal and 
external data and hardware interfaces, etc. 

5. Critique of alternative designs or approaches 

6. Major software components — For each subsystem or major 
functional breakdown (in each processing mode) , 

a. High-level diagrams of subsystems--Internal and 
external data and hardware interfaces, etc. 

b. High-level input and output specifications, in- 
cluding frequency and volume 

c. Functional baseline diagrams (treecharts) expanded 
to tv^o levels below the subsystem driver, showing 
interfaces, dati .low, and how requirements are met 

d. Facsimiles of I/O graphics displays (screens) and 
printer and plotter output 

e. Error processing and recovery strategy 

7. Hardware interfaces 

8. Internal data sets 

a. Format and description of items in header, data, 
and other records 

b. File structure (blocking and access methods) 

c. Storage requirements in all processing modes 

9. Summary of existing code that may be reused 
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10. Design team assessment 

a. List of constraints and their effects on design 

b. List of assumptions and possible effects on design 
if they are v/rong 

c. , List of concerns and problem areas — Deterrents of 

progress 

d. List of TDD requirements and an assessment of 
their impact on system size, required effort, 
cost, and schedule 

e. List of priority areas 

11. Estimates of system size, required effort, cost, and 

schedule 

a. Sizing and resources for major system components 
or subsystems — Number of modules, source lines of 
code, computer hours, effort units, etc. 

b. Life cycle expenditures — Time and effort breakdown 

for life cycle phases ' ; 

c. Staffing plan — Allocation of personnel by type for 
life cycle phases 

12. Resource allocation and external support 

a. Summary .of how system functions will be performed, 
i.e., by hardware, firmware, software, or human 

b. Rationale for selecting computers, e.g., speed, 
memory, storage, and reliability of mainframes, 
minicomputers, or microcomputers 

c. Summary of what the development team will do and 
what they need to do it, e.g., analysis support, 
librarian support, computer access and informa- 
tion, support documentation, interface access, 
integration support 
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13. Development management plan 

a. Life cycle phases and products produced 

b. Methodologies used by phase 

c. Models and tools used by phase 

d. Configuration control approach follov/ed by phase-- 

1 

Controlled itemSf forms, procedures, approval, au- 
thorization, distribution 

14. Personnel organization and interfaces — Diagram shov;ing 
organizational interfaces, their points of contact, and 
their responsibilities 

15. Testing strategy 

a. General approach to testing (methods) 

b. Extent — Responsibility and procedures for unit 
tests, build/release tests integration tests, sys- 
tem tests, acceptance tests 

c. Control mechanisms — Internal, external, and qual- 
ity assurance review procedures; approval; author- 
ization; configuration integrity procedures 

16. Issues, TBD items, and problems — A characterization of 
all those things that affect plans for detailed design, 
an assessment of their effect on progress, and a course 
of action to resolve them, including required effort, 
schedule, and cost 

17. Milestones and schedules — Including delivery of inter- 
faces and externally developed software for integration 
and testing 
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CDS? PBESERITATSeRS K3ATEK 


© Siitmductjon end Assnda 
© E2es!0H OvcyvsGvy 

O t«^ jgh-Loi»6S Ojesrams ai Operating Gcsnarlos 
o Hlgh-f^vc! Diagrams of System Structura 
O r</3ajor Software Componants 

— MisSt-Leuel Diagrams of Sybsyctsms 
— High-Lovol B/0 Spaciflcationo end intesfccos 

— runctiona! Bnsdino Diagrams (Treschartsi 
— Error Procosafng and Rocowsry Strotogy 

— ncstrictlons of Processing £¥3odoa 
— Intcmei Sscroga Ro^qulramcnts 

O Design Team Assossmont 
O tmpiomentatien Strategy end Trocoabliity 
O System Size, Required Effort, Cost, end Sciisduls Estimates 
O Resource AUccation end Entemc! Support 
o Dovciopment rtfienegoment Pian 
O Porconno! Orgenfeatien and interfaces 
O Testing Strategy 
O fs3U33, TBD items, and Problems 
O illtiestonoo end Schoduios 


CBR 


PRESEPJTERS 

Software Dovefopmont Team 

PARTICIPAASTS 

Roquiromonts Definition Team 

Quality Assurance Representatives From Both 
Groups 

Customer Interfaces for Both Groups 

TIP^E 

After Dotailod Design Completed and Oeforo 
Implomentatfon Started 

HARDCOPY 

DISTRlBUTIORi 

iViinimum of 5 Days Before CDR 
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1. Er.trcduciiian 

2. Doa’gn Ovarvlaw 

3. HEgSi-Lovc! Blagrcms of Opsrating EceoQftos 

4. f-55gli-Lovo! EJScsrantjs of StrtJsturcs 

5. r»^sjcr Softwara Componcnte 

Q. £^5gh-Lcvc5 OJagrams of Subsysksms 

b. H5g!i-Lovc5 i/0 Spcci»lcatfoMS and InScrfacos 

c. functional Bosoiino DJcgrams (Tresclisirte} 

d. Error Procossiing end Recovery Strategy 
o. Flcotrictlons of Proccsolng P/Jodoo 

f. Interna! Stcrago Roquircmctita 

g. Botciled i/0 S;pociftcstlono 

ID Pro cossing Centro! PorErnGtora 
(2) Scroon, Printer, and Ptottcr Permsts 
0. Wordwero Intorfccoo ; 

7. Interna! Data Sot Dofinitlons 

8. Reusabio Cede Summary ' 

9. Design Tocm Aonnsamont 

W. ImpSemontotion Strategy and Trecoabiiity 

11. Sy.etom Sizo, Required Effort, Cost, nnd Schedule Estimates 

12. Rcsourco Allocation and Eutornai Su ?port 

13. Dovolopmcnt IViancgGmont Plon 

a. Life Cyc!o and Products 

b. IVIothcdologios 

c. IViodeis and Tools 

d. Configuration Control Approach 

14. Porsonnol Organization and Interfacss 

15. Testing Strategy 

a. Goneral Approach 

b. Extent 

c. Control r/iochnnisms 

16. Issues, TBD Items, end Problems 

17. iVlilestcnes and Schoduiss 
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CDR 


The CDR hardcopy material contains 

1. Introduction — Purpose of the system and outline of re- 
view material 

a. Requirements summary--Or igin and format of re- 
quirements; list of major system components, in- 
cluding the top-level (basic) requirements which 
they satisfy and which are covered in the review 

b. Additional derived software requirements — Require- 
ments derived by the development team during the 
requirements analysis phase 

(1) Operating scenario requirements — Data han- 
dling, e::ecution frequency, turnaround time, 
checkpoint/restart capabilities, graphics 
needs, etc. 

(2) Environment considerations — Target computing 
machine, operating system, computer system 
support software, graphics packages and hard- 

t 

ware, etc. 

(3) Softv/are legacy (past experiences and history) 

(a) Cost factors — Experience history, design 
models, reusable code, etc. 

(b) Schedule factors--Deadlines; hardware, 
software, and support dependencies; etc. 

2. Design overview 

a. Requirements summary--List cross-referencing top- 
level (basic) requirements to major system compo- 
nents presented at SRR 

b. Performance requirements--Cross-reference list of 
performance requirements that led to partitioning 
of system into major components 
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c. Design drivers — Primary factors that influenced the 
development team's design, e.g., operating 
scenarios, environmental considerations, and soft- 
v;are legacy 

3. High-level diagrams of operating scenar ios--Input stim- 
ulus, processing, output stimulus, and interfaces to 
show how requirements are met 

4. High-level diagrams of system structure — Internal and 
external data and hardware interfaces, etc. 

5. Major software components — For each subsystem or major 
functional breakdown (in each processing mode) , 

a. High-level diagrams of subsystems--Internal and 
external data and hardware interfaces, etc. 

b. High-level input and output specifications, in- 
cluding frequency and volume 

c. Baseline diagrams (treecharts) expanded to the sub- 
routine level, showing interfaces, data flow, in- 
teractive control, interactive input and output, 
and how requirements are met 

d. Error processing and recovery strategy 

e. Restrictions in each processing mode 

f. Internal storage requiremer.ts--Descr iption of ar- 
rays, their size, their data capacity in ell proc- 
essing modes, and implied limitations of processing 

g. Detailed input and output specifications 

(1) Processing control parameters--NAMELISTs, etc. 

(2) Facsimiles of I/O graphics displays (screens) 
and printer and plotter output 
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6. Hardware interfaces 

7. Internal data sets 

a. Format and description of items in header, data, 
and other records 

b. Pile structure (blocking and access methods) 

c. Storage requirements in all processing modes 

8. Summary of existing code that may be reused 

9. Design team assessment 

a. List of constraints and their effects on design 

b. List of assumptions and possible effects on design 
if they are wrong 

c. List of concerns and problem areas~-Deterrents of 
progress 

t 

d. ' List of TBD requirements and an assessment of 

their impact on system size, required effort, 
cost, and schedule 

e. List of priority areas 

10. Implementation strategy and traceability 

a. 


b. 

c. 


Build/release overview and schedule, indicating 
establishment of internal and external data inter- 
faces for both connection tests and data flow 
tests and showing delivery of interfaces and ex- 
ternally developed software 

Build/release capabilities--List of capabilities 
implemented in each build/release by subsystem 

Requirements traceability--Cross~reference list of 
build/release capabilities to basic and derived 
software requirements 


A-21 
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Estimates of system size, required effort, cost, and 
schedule 


, a. Sizing and resources for major system components or 
subsystems — Number of modules, source lines of 
' . code, computer hours, effort units, etc. 

b. Life cycle expenditures--rime and effort breakdov;n 
for life cycle phases 

c. Staffing plan — Allocation of personnel by type for 
life cycle phases 

12. Resource allocation and external support 

a. Summary of how system functions will be performed, 
i.e., by hardware, firmware, software, or human 

b. Rationale for selecting computers, e.g., speed, 
memory, storage, and reliability of mainframes, 
minicomputers, or microcomputers; 

c. Summary of what the development team will do and 
what they need to do it, e.g., analysis support, 
librarian support, computer access and information, 
support documentation, interface access, integra- 
tion support 

13. Development management plan 

a. Life cycle phases and products produced 

b. Methodologies used by phase 

c. Models and tools used by phase 

d. Configuration control approach followed by phase — 
Controlled items, forms, procedures, approval, au- 
thorization, distribution 

14. Personnel organization and interfaces — Diagram showing 

organizational interfaces, their points of contact, and 

their responsibilities 
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15. Testing strategy ' 

1 

a. ' General approach to testing (methods) 

b. Extent--Respohsibility and procedures for unit 
tests, build/release tests, integration tests, 
system tests, acceptance tests 

c. Control mechanisms — Internal, external, and qual- 
ity assurance review procedures; approval; author- 
ization; configuration integrity procedures 

16. Issues, TBD items, and problems--A characterization of 
all those things that affect plans for detailed design, 
an assessment of their effect on progress, and a course 
of action to resolve them, including required effort, 
schedule, and cost 

17. Milestones and schedules--Including delivery of inter- 
faces and externally developed software for integration 
and testing 
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O introdisctson cn<d Againda 
O System RfKiU57QmGnts Sisssimcry 
6 Anaiycss OvQrvtow 
,© Support System Ovorviavv 

— EViaJor Softwara Gamporsonte 
— System Tcstsing Philosophy 

— Syoterrj Testing end Porfcrmanco [^valuation Rosuito 
— RequIrernGnts Verification Philosophy 
— System ScfCn/aro end Dcoumsrstaticn Status 
O Opsrations end Support Plan 

— pGiconne! Asoignmonts and fiosponoibiSitsos 
— Os’gonizaCienE! Entcrfacos 
~ Data Avallabiilty , 

— FeeiJiStos 

— Cporatino Scenarios 
© System relanagomont Plan 
© Porsonnol Ofgcnizatlon and Interfaces 
O Issues, TBD Items, and Problems 
© Contingoncy Plans 
o l^ileotonos and Timeline of Events 



ORR FORMAT 

PRESENTERS 

Operations and Support Team 

PARTICIPANTS 

User Accoptanco Test Team 

TimE 

Requirements Definition, Softv^aro 
Oovolopmont, end Software iVlaintonsrica 
Roprosentatives 

Quality Assuranca Representatives From Ail 
Groups 

Customer Interfaces for Ail Groups 
After Acceptance Testing Completed and SO 

HARDCOPY 

Days Before Operations Start 
iVlinimum of 5 Days Before ORR 

DISTRIBUTION 
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1. Entrcducticn 

2. SyotQsir? Rciiuiromcnto Summary 

3. AnaSycia Overvistw 

4. Support System OvcrvJQW ■ 

Q. [yJajer SofavtfOfa Compononto 

b. System Tosthtg PhiSosophy 

c. System Testing and Porfcrmanco Evaluation RosuSto 

d. Requiromento Ucrification FhHosophy 

o. System Softivam and Occumontation Status 

5. Oparaiiona end Support PJon 

a. Personnsi Assignments and RosponsibiUtlos 

b. Organisational Sntorfecos 

c. Data AvallabiSity 

d. FccISitlos 

(1) i\3arma! Oparctlono 

(2) Crltlcci Operations 

(3) Emergency Operations 

(4) Contingency Operations 
o. Operating Scenarios 

(1) Support Kequirsmonto 

(2) Timeline of Events 

(3) Operating Procoduros 

(4) Resources Required 

6. System H/ianagomont Plan 

a. Personnel Assignments 

b. Description of Required Products 

c. Configuration Control Approach 

d. Enhancement/iVlaintonanco Procedures 

0 . Reporting and Testing EvaSuation Proceduros 
f. System Porformcnco Evaluation Procedures 

7. Personnel Organization and Interfaces 

8. issues, TBD items, and Probiems 

9. Contingency Plans 

10. Milestones and Timeline of Events 







ORR 


The ORR hardcopy material contains 

1. Introduction--Purpose of the system and outline of re- 
view material 

2. System requirements summary- -Review of top-level (basic) 
requirements 

a. Background of requirements--Overviov; of project 
characteristics, major events, and support 

b. Derivation of requirements- -Diagram showing 

( 1 ) Project office input 

(2) Support organization input 

(3) System engineering input 

(4) Software engineering input 

c. Type of requirements 

(1) Evaluation of support requirements 

(a) Typical support 

(b) Critical support 

(c) Special support < 

(d) Contingency support 

(2) Operational support scenarios 

(3) Relationship of requirements matrix, e.g., 

. relationship of top-level requirements to op- 

erational support scenarios 

d. Constraints, assumptions, and guidelines 

(1) Organizational interf aces--Organizations that 
provide system and support input and receive 
system output 

(2) Data availability for the operating sce- 
narios--Frequency , volume, format 

(3) Facilities — Computing hardware, environment 
characteristics, communications protocols, etc. 
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(4) General system considerations--High-level de- 
scription of computer storage, graphics, and 
failure/irecovery requirements; operator inter- 
action requirements; system error recovery and 
diagnostic output requirements; etc. 

(5) Support and test softvjare considerations-- 
High-level description of requirements for 
data simulators, test programs, and support 
utilities 

3. Analysis overview- -Summary of all analysis leading to an 

operational system 

a. Introduction--Project overview, support firsts, 
bases for analysis 

b. Major areas of analysis and findings 

c. Special studies and results 

4. Support system overview 

a. Major software components--Purpose, general charac- 
teristics, and operating scenarios supported by 
programs and subsystems 

b. System testing philosophy for each type of test- 
ing--Unit, build/release, integration, checkout, 
and acceptance 

c. System testing and performance evaluation results-- 
Summary of test results and evaluation of system 
performance measured against performance require- 
ments 

d. Requirements verification philosophy--Demonstration 
of methods used to ensure that the software satis- 
fies all system requirements 
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e. System softvjare and documentation status-“Summary 
of completed work packages and list of incomplete 
work packages with scheduled completion dates and 
explanation of delays 

5. Operations and support plan 

a. Personnel assignments and responsibilities--Opera- 
tions and support team organization, key personnel 
and their responsibilities, etc. 

b. Organizational interfaces — Diagrams and tables in- 
dicating organizational interfaces, their points of 
contact and their responsibilities; data flow and 
medium (forms, tapes, voice, log) 

c. Data availability — Nominal schedule of input and • 
output data by type, format, frequency, volume, 
response time, turnaround time 

d. Facilities — Nominal schedule of accessibility to 
computers, support hardware, special-purpose hard- 
ware, operating systems, and support software for 

(1) Normal operations 

(2) Critical operations 

(3) Emergency operations 

\ 

(4) Contingency operations 

e. Operating scenarios--Step-by-step operating proce- 
dures, including personnel assignments, points of 
contact for decisions, authorization, and external 
interfaces, timelines, contingencies, and emergen- 
cies 

(1) Support requirements--What has to be done? 

(2) Timeline of events--When do things get done? 

(3) Operating procedures--How do things get done? 
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(4) Resources required for operations--What is 

necessary -to get. things done? 

(a) Hardware--CPUs, disks, tapes, printers, 
graphic devices, etc. 

(b) Memory considerations--Program s' •>rage, 
array storage, data set buffers, etc. 

(c) Timing considerations--CPU and wallclock 
time in terms of samples and cycles proc- 
essed and I/O time in terms of data sets 
used and type of processing 

(d) Peripheral space considerations--Data 
storage and printout 

(e) Staffing considerations--Number and type 
of people in terms of processing steps 

' and shifts 

(f) Physical space considerations--Desks, 
chairs, shelves, storage, etc. 

6. System management plan 

a. Personnel assignments--Operations and support team 
organization; key personnel and their responsibili- 
ties and start dates; etc. 

b. Description of required products--Name , form, and 
contents of products; production standards; date 
scheduled for delivery; name of organization (per- 
son) responsible; internal, external, and quality 
assurance review procedures; authorization proce- 
dures for release; etc. 

c. Configuration control procedures--Explanation of 
step-by-step procedures for maintaining system in- 
tegrity, recovering from loss, fixing faults, and 
enhancing system 
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d. EnhancGraGnt/maintenancs procedures--Initiation, 
forms, reviews, approval, authorization 

e. Reporting and testing evaluation procedures-- 
Forms, reviews, approval, authorization, distribu- 
tion 

f. System performance evaluation procedures--Approach 
for ongoing evaluation of system performance 

7. Personnel organization and interf aces--Diagram showing 
organizational interfaces, their points of contact, and 
their responsibilities 

8. Issues, TBD items, and problems--A characterization of 
all those things that affect intended normal operations 
as perceived by the developers and users, an assessment 
of their effect on operations, and a course of action 
to resolve them, including required effort, schedule, 
and cost 

9. Contingency plans — Prioritized list of things that 
could prevent normal operations, including the steps 
necessary to work around the problems, the defined 
levels of operations during the workarounds, and the 
procedures to attempt to regain normal operations 

10. Milestones and timeline of events--Diagrams, tables, 
and scripts of events; operating scenarios; mainte- 
nance; enhancement; reviews; training; etc. 
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This appendix presents suggested contents and formats for 
various documents produced during the software development 
life cycle. Following each document format summary is a 
brief description of the contents. 



1. IrStE-GdtECtEOn 

2. PFobiem Staternsrst 
2. Apjjrcachi 


a. TcsiisnSGal 
h. EllSastagsmsmt 
c. ConfEgisFatlors CoratroS 

4. Eiescureo Estsmatas and Staf?!ng Frofis© 

5. Schedulos ai^d rt^iSestones 

6. Stems Ressusred From tSia Customsr 

7. L’tems To Ba DeSivared to the Customer 
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Software Development Plan 

The software development plan (see Reference 2) is completed 
during the requirements. -analysis and preliminary design 
phases. It contains 

1. Introduction, including purpose, background, origin of 
requirements, personnel organisation, and summary of plan 

2. Problem statement, i.e., v/hat has to be done and what 
steps are necessary, including what is different about 
this project compared v/ith a typical development project 

3. Approach 

a. Technical approach (by life cycle phase) 

(1) Assumptions and constraints 

(2) Anticipated and unresolved problems 

(3) Major activities, including methods and tools 
used 

(4) Products produced, including contents and 
standards for production 

b. Management approach (by life cycle phase) 

(1) Concerns 

(2) Resource estimates, including system size in 
terms of code origin and language as well as 
support ?uch as project personnel, computer 
time, external groups, and training 

(3) Personnel assignments and schedules 

(4) Progress accounting procedures, i.e.. data 
collection, progress measurement, and progress 
reporting methods 

(5) Quality assurance procedures 

(6) Internal and exteriial review methods 
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’ /, ■ Software Development Plan 

c. Configuration management procedures for maintaining 
requirements, design, code, and documentation in- 
tegrity 

4. Summary of resources required and cost 

5. Milestone charts showing the development life cycle, 
delivery of required external interfaces, scheduled in- 
tegration of externally developed software, delivery of 
required information, delivery of development products, 
and scheduled reviews 

6. List of items (dated) required from the customer 

7. List of items (dated) to be delivered to the customer 
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1. BoCCS’OJStECn Q?SCS List GSf r^a|s? Comj2OTt0rEtG 

2. JCey Peirsons^©l ©?is^ Tltsir l^eGjsosxsjbifBSiGa 

3. OessrEpticsTi g? CapsbiSities in Eash BiEiisS/UcSeaso 

4. OsSsworalsSa Products 

5. Msstory of Events, Schedules, and rJSiSestones 

G. I^istory of Systenj Size, Elecfulred Effort, and Cost 
Estimates 

7. S-Ssstory of Source Coda Changes 

8. ffestory of Accomplishments 

9. E^sstory of Outstanding TBD items. Changes, Errors, 
and ProoSems 

10. Msstory of Verification of System Requirements 
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Project Notebook 


The project notebook is started at the beginning of the 

project and maintained throughout the development life 

cycle. It contains i 

1. Description and list of major components 

a. Description of project 

b. List of major components of the system 

2. Key personnel and their responsibilities 

a. Name, title, organization, telephone number, start 
date, and responsibility of key personnel 

b. Name, title, organization, telephone number, start 
date, and responsibility of points of contact for 
interfacing groups 

3. Description of functional capabilities in each build/ 

release listed in relation to the requirements 

4. Deliverable products 

a. Name of product, form of product, date scheduled 

for delivery, and actual date delivered for each 
product ' 

b. Name of person responsible for each product 

5. History of events, schedules, and milestones 

a. Milestone chart identifying tasks and originally 
scheduled completion dates 

b. Biweekly updates to milestone charts 

6. History of system size, required effort, and cost esti- 
mates 

a. Graph of number of modules in operational version 

of system versus time in weeks, including estimates 
of final number in end product 
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Project Notebook 





b. Graphs of number of source lines of code (with and 
without comments) and executable linos of code in 
operational version of system versus time in weeks, 
including estimates of final numbers in end product 

c. Graph of staff-months of effort expended versus 
time in weeks, including estimates of planned ex- 

' penditures (weekly totals and accumulated) 

7. History of source code changes--Graph of number of 
source code changes and change reports versus time in 
weeks for each development life cycle phase and overall 

8. History of accomplishments--Monthly list of achieved 
milestones, completed tasks, and delivered products 

9. History of outstanding TUD items, changes, errors and 
problems, i.e., list of those things that are unresolved 

10. History of verification of system requirements, i.e., 
list of functional capabilities that have entered the 
system and were tested by build/release, system test- 
ing, and acceptance testing with date of test and test 
resul t 
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Requirements Analysis .Siimmarv, Report 

, , ' J 

The requirements analysis sum.iviary report is completed at the 
end of the requirements analysis phase. It contains 

1. Introduction/ including purpose and background of project 

a. Overall system concepts 

b. Discussion and high-level pictures (diagrams) of 
system showing hardware interfaces, external data 
interfaces, and data flov/ 

c. Discussion and high-level pictures (diagrams) of 
operating scenarios with interfaces and data flow 

2. System constraints 

a. Hardware availability 

(1) Execution 

(2) Storage 

(3) Peripherals 

b. Operating system limitations 

I 

c. Support software limitations 

3. Development assumptions ; 

4. Arens of concern and TBD requirements 

a. List of concerns and problem areas, i.e., deter- 
rents of progress 

b. List of TBD requirements and an assessment of their 
impact on system size, required effort, cost, and 
schedule 

c. List of priority areas 

5. Analysis of basic and derived requirements for system, 
including level of importance of key issues and com- 
pleteness 
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Requirejncnts Analysis Summary Report 


a. Stimulus for input 


(1) 

Frequency *; 


(2) 

Volume 


(3) 

Coordinates and 

units 

(4) 

Timing 


Processing 


(1) 

Functionality 


(2) 

Accuracy 


(3) 

Timing 


(4) 

Error handling 


Stimulus for output 


(1) 

Frequency 


(2) 

Volume 


(3) 

Coordinates and 

units 

(4) 

Timing 



6. Analysis of basic and derived requirements for subsys- 
tems or major functional breakdowns, including level of 
importance of key issues and completeness. Same as for 
item 5 above except for subsystems or major functional 
breakdown 

7. Data interfaces — For each interface, 

a. Description, including name, function, frequency, 
coordinates, units, and computer type, length, and 
representation 

b. Format 

(1) Organization, e.g,, physical sequential 

(2) Transfer medium, e.g., 9-track tape, printout 
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Requicements Analysis Summary Report 

(3) Layout of frames, samples, records, blocks, 

, and/or transmissions 

(4) Storage requirements 

8. Summary of existing code that may be reused 

'9. Estimates of system size, required effort, cost, and 
schedule 
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Q. Gi’era-il ByctQm Sos^sspto 

b. E*2lg3x-LQvo2 Psstsires of System 

c. Opsratlsig ScsKurios 

d. Oosags^ Stafys 

©. Ca’itlj|UQ of /VEtersisa’iisv© Designs 

2. Sobsystoms 

a. E-lsgh-LGUQS Plctyros 

b. DQsonptiosi of inpsst and Oytpyt 

c. DGccriptBon of ProcessEng 

d. FynotlcmaS.B-asQilno Diagrams 

e. ProEogs and PDL for First LevoS 

S. Kosoyreo Keqyimmsnte 

a. f-tordware 

b. Date Definitions 

c. Peripbera! Spaca Considerations 

d. rJIcmory Considerations 

4. Data interfaces 

a. Doscrtptlon 

b. E-ormat 

5. Sunrumary of R(5usab!e Software 
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Preliminary Design Report 

The preliminary design report is completed at the end of the 
preliminary design phase. It contains 

1. Introduction, including purpose and background of project 

a. Overall system concepts 

b. Discussion and high-level pictures (diagrams) of 
system showing hardware interfaces, external data 
interfaces, and data flov; 

c. Discussion and high-level pictures (diagrams) of 
operating scenarios with interfaces and data flow 

d. Design status 

(1) List of constraints and their effects on design 

(2) List of assumptions and possible effects on 
design if they are wrong 

(3) List of concerns and problem areas, i.e., de- 
terrents of progress 

(4) List of TBD requirements and an assessment of 
their impact on system size, required effort, 
cost, and schedule 

(5) List of priority areas 

e. Critique of alternative designs 

2. For each subsystem or major functional breakdown, 

a. Discussion and high-level pictures (diagrams) of 
subsystem, including interfaces, data flow, and 
communications for each processing mode 

b. High-level description of input and output 

c. High-level description of processing keyed to 
operator-specified input and actions in terms of 
points of control, functions performed, and results 


B-16 


9108 




Preliminary Design Report 


obtained (both normal and abnormal, i.e., error 
processing and recovery) 

d. Functional baseline diagrams (treecharts) expanded 
to two levels below the subsystem driver 

e. Prologs^ and PDL for each module through first 
level below subsystem driver 

3. Resource requirements--Discussion, high-level pictures, 
and tables for system and subsystem 

a. Hardv/are 

b. Data definitions, i.e., data groupings and names 

c. Peripheral space considerations 

(1) Data storage 

(2) Printout 

d. Memory considerations 

(1) Program storage 

(2) Array storage 

(3) Data set buffers 

4. Data interf aces--For each internal and external inter- 
face, 

a. Description, including name, function, frequency, 
coordinates, units, and computer type, length, and 
representation 

b. Format 

(1) Organization, e.g., physical sequential 

(2) Transfer medium, e.g., tape 


^Module comments to describe the module's purpose, opera- 
tion, calling sequence arguments, external references, etc. 
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Preliminary Design Report 

I 

(3) Layout of frames, samples, records, 
and/or transmissions 

, ; 

(4) Storage requirements 

5. Summary of existing code that may be reused 
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Detailed Design Document 


obtained (both normal and abnormal, i»e., error 
processing and recovery) 

e. Baseline diagrams (treecharts) expanded to the 
subroutine level shov/ing interfaces, data flow, 
interactive control, interactive input and output, 
and hardcopy output 

f. Restrictions in each processing mode 

g. Internal storage requirements, i.e., description of 
arrays, their size, their data capacity in all 
processing modes, and implied limitations of proc- 
essing 

h. Detailed input and output specifications 

(1) Processing control parameters, e.g., NAI-IELISTs 

(2) Facsimiles of graphic displays for interactive 
graphic systems 

(3) Facsimiles of hardcopy output 

i. List of numbered error messages with description of 
system's and user's actions 

j. Description of COMMON areas 

k. Prologs^ and PDL for each subroutine 

3. Resource requireraents--Discussion, high-level pictures, 
and tables for system and subsystems 

a. Hardware 

b. Data definitions, i.e., data groupings and names 

c. Peripheral space considerations 

^ (1) Data storage 

(2) Printout 


^Module comment to describe the module's purpose, opera- 
tion, calling sequence arguments, external references, etc. 
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Detailed Design Docvinu^nt'- ’ 

d. Memory considerations 

(1) Program storage 

(2) Array storage 

I . • ‘ 

(3) Data set buffers 

4. Data interfaces — For each internal and external inter- 
face, 

a. Description, including name, function, frequency, 
coordinates, units, and computer typo, length, and 
representation 

b. Format 

(1) Organization, e.g., direct access 

(2) Transfer medium, e.g., disk 

(3) Layout of frames, samples, records, blocks, 
and/or transmissions 

(4) Storage requirements 

5. Summary of existing code that may be reused, including 
list of code with level of modification 
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1. 

a. Fjirjscssa 

b. Typo SEid Ls^ieS ©f TestEitg 

c. Sshedu§Q 

2. Far Eocl^ T©st 

Q. PlE?pOS9 

fo. OstosSsd Spscafsjsatlon o1? Es^pist 

c. Keqiiirod ERvimsurnsrs® 

d. Oparst'lGJis! Frocedinro 

©. CofiaHsd SpccIfj'sstlGn of Oisfput 

f. Fass/FasS Cpitsyia 

g. Of'rcicGSson of Rcsolts 
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Test Plans 


A test plan is completed before the period in which it will 
be uscdf with sufficient time for testers to review it. A .. 
test plan contains ’ ‘ \ ■ 

1. Introduction/ including purpose, type and level of test- 
ing, and schedule 

2. For each test,* 

a. Purpose of test, i.e., specific capabilities or 
requirements tested 

b. Detailed specification of input 

c. Required environment, e.g., data sets required, 
computer hardv;are necessary 

d. Operational procedure, i.e., how to do the test 

e. Detailed specification of, output, i.e., the ex- 
pected results 

f. Criteria for determining whether or not test re- 
sults are acceptable 

g. Section for discussion of results, i.e., for ex- 
plaining deviations from expected results and iden- 
^ifing cause of deviation or for justifying 
deviation 



1 . 

a. ©vGraii System Cog’asspts 

b. i's8s5ii-L©uG!l Pictes'GS 

c. Opsrstssig Seersas’ios 

2 . 

a. Gvas’aii Capability 

b. Assymptlons arbd ilGStriaticsiia 

c. E^?gSt-L©VGi PictiiFes 

d. Gosariptian inpist ssid Gsitpsit 
Q. Goccriptlori of ProcessiJig 

3. PeqBjipGmGrsts for Eiiecutloc^ 

a. RosoiUErcos 

b. Run information 

c. Control Paramstor information 

4. Datalicd Description of input and Output 

a. Facsimiles of Graphic Displays 

b. Facsimiles of Hardcopy Output 

c. List of iViOssages 
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User's Guido 

Formalization of the user's guide is started during the 
implementation phase using the design document as a starting 
point. Reorganization is completed for the beginning of 
system testing, and a typed draft is completed for the be- 
ginning of acceptance testing. The user's guide is com- 
pleted by the end of acceptance testing. It contains 

1. Introduction, including purpose and background 

a. Overall system concepts 

b. Discussion and high-level pictures (diagrams) of 

system showing hardware interfaces, external data 
interfaces, and data flow 

c. Discussion and high-level pictures (diagrams) of 
operating scenarios with interfaces and data flow 

2. For each subsystem. or major functional breakdown, 

a. Overall subsystem capability 

b. Assumptions and restrictions to processing 

c. Discussion and high-level pictures (diagrams) of 

subsystems, including interfaces, data flow, and 
communications for each processing mode 

d. High-level description of input and output 

e. Detailed description of processing keyed to 
operator-specified input and actions in terms of 
points of control, functions performed, and results 
obtained (both normal and abnormal, i.e., error 
processing and recovery) 
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User's Guide 


3. Requirements for execution 

a. Resources — Discussion, high-level pictures (dia- 

grams) , and tables f<.r! system and subsystems 

(1) Hardware 

(2) Data definitions, i.e., data groupings and 
names 

(3) Peripheral space considerations 

(a) Data storage 

(b) Printout 

(4) Memory considerations 


(a) 

Program storage 



(b) 

Array storage 



(c) 

Data set buffers 



Timing considerations 



(a) 

CPU tim.e in terms 
processed 

of 

samples and cycles 

(b) 

I/O time in terms 

of 

data sets used and 


type of processing 


(c) 

Wa 11c lock time in 

terms of samples and 


cycles processed 

b. Run information--Cont>:ol statements for various 
processing modes 

c. Control parameter informat ion--By subsystem, de- 
tailed description of all control parameters (e.g., 
NAMELISTS), including name, computer type, length, 
and representation, and description of parameter 
with valid values, default value, units, and re- 
lationship to other parameters 


B-27 



User's Guide 


4. Detailed description of input and output by system and 

subsystem 

a. Facsimiles of graphic displays Cor interactive 
graphic systems in the order in which they appear 
for each processing mode 

b. Facsimiles of hardcopy output in the order in which 
it is produced, annotated to show v/hat parameters 
control it 

c. List of numbered messages with ex.planation of sys- 
tem's and user's actions annotated to show subrou- 
tines that issue the message 
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lSES©R0PTO®a^ 


1. iretmdi^GtsoTi 

2. Subsystems 

a. CvcraiS C£3j3Ebl?!ty 

b. AssissvsptEOS^r. estd O^CEtpactlorts 

c. E’^cgb-LovcS Ptctofcs 

d. OoEcroptian of input and Output 
Q. EasoSino Diagrams' 

3. Baquiromsmts for C-Foation ' 

a. ElesGurcos 

b. Croatioira Ersfcrmatlon , ; 

c. Program Structuro Bnfcrmatiors ' 

4. OotaiSed Description of input and Output 

5. SnternaS Storage Requlromonts 

6. Data interfaces 

7. Description of CORfJiViOW Areas 

8. Prologs and PDL 

9. List of Software from Support Libraries 










System Descr iption ■ , .’.Yi -,. , : 

Formalization of the system description is started at the 
beginning of system ;-testing using the design document as a 
starting point. It is completed by the end of acceptance 
testing. The system description contains 

1. Introduction, including purpose and background of project 

a. - Overall system capabilities 

b. Discussion and high-level pictures (diagrams) of 
system showing hardware interfaces, external data 
interfaces, and data flow 

c. Discussion and high-level pictures (diagrams) of 
operating scenarios with interfaces and data flow 

2. For each subsystem or major functional breakdown, 

a. Overall subsystem capability 

b. Assumptions and restrictions to processing 

c. Discussion and high-level pictures of subsystem, 
including interfaces, data flow, and communications 
for each processing mode 

d. High-level description of input and output 

e. Detailed baseline diagram at subroutine level show- 
ing interfaces, data flow, interactive control, 
interactive input and output, including messages, 
and hardcopy output 

3. Requirements for creation 

a. Resources--Discussion, high-level pictures (dia- 
grams) , and tables for system and subsystems 

(1) Hardware 

(2) Supoort data sets 


D-30 


9108 




System Description 


(3) Peripheral space considerations 

(a) Source code storage 

(b) Scratch space 

(c) Printout 

(4) Memory considerations 

(a) Program generation storage 

(b) Data set buffers 

(5) Timing considerations 

(a) CPU time in terms of compile, build, and 
execute benchmark test 

(b) I/O time in terms of the steps to create 
the system 

b. Creation information--Control statements for 
various steps 

c. Program structure inf ormation--Control statements, 
e.g. , for overlay or for addressing and loading 

4. Detailed description of input and output by step 

a. Source code libraries for. system and subsystems 

b. Object code libraries 

c. Execution code libraries 

d. Auxiliary libraries, e.g., support tables 

5. Internal storage requirements, i.e., description of 
arrays, their size, their data capacity in all process- 
ing modes, and implied limitations of processing 

6. Data interf aces--For each internal and external inter- 
face, 

a. . Description, including name, function, frequency, 
coordinates, units, and computer type, length, and 
representation 
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System Description • ' 

b. Format J, ' 

(1) Organization, e.g., indexed 

(2) Transfer medium, e.g., drum 

(3) Layout of frames, samples, records, blocks, 
and/or transmissions 

(4) Storage requirements 

7. Description of COMMON areas 

8. Prologs and PDL for each subroutine (usually produced in 
a separate volume) 

9. Alphabetical list of subroutines from support data sets, 
including — for each subroutine — a reference to the sup- 
port data set from which it comes and a description of 
the subroutine's function 
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Software Development History, 


The softv;are development history is completed v/ithin 3 months 
of software acceptance.. , It contains 

1. Project description and background 

a. Problem statement and list of key requirements 

b. Origin of requirements 

c. Customer of system 

d. Purpose of system- 

e. Key dates (actuals) 

(1) Availability of functional specifications and 
requirements 

(2) Development phase dates (start and finish) , 
i.e., requirements analysis, preliminary de- 
sign, detailed design, implementation, system 
testing, acceptance testing 

(3) Event dates, e.g.', SRR, PDR, CDR, ORR 

f. Key products produced 

(1) All software, i.e., the system, simulators, 
and test programs 

(2) All documents, i.e., development plan, project 
notebook, requirements analysis summary re- 
port, preliminary design report, detailed de- 
sign document, test plans and results, user's 
guide, and system description 

g. Development organization 

h. System characteristics 

(1) Total, new, and reused number of source lines 
of code (with and without comments) of end 
product 
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Software Development History 

(2) Total, nev/, and reused number of modules of 
! ‘ end oroduct 

J ■ 

j (3) Total/ managorial, programmer, and support 

j service hours required for development 

, 2. Development history 

' a. Original and updated estimates of system size, re- 

. quired effort, schedule, and cost 

' b. Organizational structure and key personnel 

] c. Specified approaches, e.g., methods, practices, 

standards, and tools 

r 

f d. Unique approaches, e.g., independent verification 

and validation team, prototyping 

I e. Target development machine and programming languages 

f. Special problems encountered ^ 

g. Build/release history 

I 

h. Test history > 

i. Configuration control start dates for requirements, 
design, and code 

3. Project assessment 

a. Substantiated major strengths of the development 
process and product 

b. Substantiated major weaknesses of the development 
process and product 

c. Major problem areas 

d. Development plan timeliness and usefulness 

e. Adherence to development plan 

f. Adherence to standards and practices 

^.3 B-35 



9108 



Software Developnier.t History, 

g. Design timeliness, completeness, and quality 

h. Code timeliness, completeness, and quality 

i. Test timeliness, completeness, and quality 

j. Personnel adequacy (number and quality) 

4. Functional specifications and requirements 

a. Origin and timeliness 

b. Completeness and adequacy for design 

c. Change history and stability 

d. Clarity (i.e., were there misinterpretations?) 

5. Summary 

a. List of shortcomings of the development process and 
product 

b. List of successful aspects of the development 
process and product 

c List of things that should be done differently for 
future projects 

d. List of things that should be done similarly for 
future projects 

e. List of the major causes of errors 

6. References 

a. List of relevant background documents, e.g., func- 
tional specifications and requirements, software 
development plan, test strategy and plans 

b. List of reports, e.g., requirements analysis sum- 
mary report, preliminary design report, detailed 
design document, test plans with results, change 
histories 

c. List of any other necessary reference material 
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enSEP EJJAr«f3PLI! QP STEPS ?CJ ORCSAKaSE A PROJECT 


Thisi .appendix cont^iina a brief example of some stepi? the de- 
velopment manager must taRe to organize a project. 

Senior-level development managers usually have, in their 
minds, the benefits of experience; unfortunately, this ex- 
perience is not usually written down. When the benefits of 
senior personnel's experience are documented, they are usu- 
ally summarized in a way that is difficult for the junior- 
level development manager to interpret and apply. Without 
similar experience, a junior-level manager v^ill often find a 
more experienced manager's experience summary to be confus- 
ing or unintelligible. 

The example here is an illustration for the more junior- 
level manauer. It is not necessarily universally applica- 
ble; however, the example Illustrates steps and factors that 
a development manager must consider when organizing a proj- 
ect. Tlie following subsections are interrelated because the 
organizat ion of the approach depends on the constraints of 
the problem; i.e., the order in which steps are taken de- 
p«Muii.- on the problem. ■ 
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C.l ESTIMATING SISE AND EFFORT 

One of the most critical aspects of organizing a development 
project is estimating the amount of software to be developed 
and the effort required to develop it. Poor estimates, 
either high or low, cause a loss of confidence in the eyes 
of the customer as well as within the development organiza- 
tion. 

In this example, the development organization develops soft- 
ware systems for a broad application of related scientific, 
data base, and data support systems. New applications 
(project types) and computing facilities (environment types) 
are introduced every few years. The development organiza- 
tion develops systems for both the old and new computing 
facilities and for the same and different, but usually re- 
lated, applications. That is, the organization not only 
builds on past experience for continuing support of its 
charter but also branches out into new aspects of its char- 
ter because of changing technology. Therefore, in general, 
a moderate amount of code is reused from project to project. 

C.1.1 SI2E OF THE SYSTEM 

At each phase of the development life cycle, the development 
organization uses essential, supplemental, and archived in- 
formation to produce an estimate of the system's size with 
acceptable and understood uncertainty limits (see Refer- 
ence 22) . Table C-1 lists some high-level information. For 
example, in this example, the development organization uses 
3040 X (number of general subsystems) to compute tlie number 
of executable lines of code (LOC) . 
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Table C-1. Effort Estimators and Uncertainty Limits 
by Phase 


End of Phase Effort Estimators 


Limits of Uncer- 
tainty as Percent 
of Estimated Effort^ 


Proproject 

Similar projects, 
general subsystems 

±100 

Requirements 

Analysis 

General subsystems 

±70 

Preliminary design 

Specific subsystems 

±50 

Detailed design 

Actual subsystems, 
modules, code, 
documentation 

±30 

Implementation 

Modules, code, 
tests, documenta- 
tion 

±12 

System testing 

Code, tests, doc- 
umentation 

±5 


Q 

Upper limit » (Effort estimate) x (1. + uncertainty in decimal) 
Lower limit = (Effort estimate) /(I. + uncertainty in decimal) 


C.1.2 EFFORT REQUIRED TO DEVELOP THE SYSTEM 

In this example, using the size estimate from any life cycle 

phase. Equation (C-1) predicts the total effort^ required 

2 

for complete development 


gTot 




(C-1) 


^Total development effort includes administrative and tech- 
nical managers, developers, and support personnel, i.e., 
secretaries, librarians, and technical publications. See 
Table C-8 on page C-16. 

2 

Complete development encompasses the software development 
life cycle phases from requirements analysis through ac- 
ceptance testing. 
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' Tot 

v/here E i£3 effort in staff-months; the f^^’s are factors 
that increase or decrease required effort because of problem 
complexity, team -experience, schedule, methodology used, and 
so on; and 

L =» (0.8N + 0.2) X (estimate of total system size (C-2) 

in thousands of executable LOG) 

where N is the fraction of newly developed and extensively 
modified code in the system size estimate. 

Basically, to start, the manager need only consider the com- 
plexity of the problem, the development team's experience, 
and the schedule. As the development organization estab- 
lishes itself, other factors (such as methodology usage) can 
be added to fine-tune the estimation process and to apply it 
to a wider range of software development problems. 

For this example. Table C-2 provides the development organi- 
zation with a simple guideline for adjusting effort with a 

\ ; 

Table C-2. Complexity Guideline® 


Project 

Type^ 

Environment 

Type° 

Effort 
Factor (fj^) 

Old 

Old 

0.45 

Old 

New 

0.65 

'New 

Old 

0.65 

New 

New 

1.00 


^Based on SEL data and other data available to the SEL. 

Application, e.g., orbit determination, data base. The 
project type is old when the development team has more than 
2 years of experience with it. 

‘^Computing environment, e.g., IBM S/370, VAX-11/780, Intel. 
The environment type is old when the development team has 
more than 2 years of experience with it. 
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complenity factor. Table C-3 provides a simple guideline 
for adjusting effort with a team ej:perience factor. 

Table C-4 provides, a simple guideline for adjusting effort 
with a schedule factor. 


Table C-3. Development Team 

Experience Guideline 

Team Years of 
Applicable Experience^ 

Effort 
Factor (f 2 > 

10 

0.5 

8 

0.6 

6 

0.8 

4 

I.O 

2 

1.4 

1 

2.5 

^Sum of products of fraction of team member participation 
with his/her years of applicable experience (requirements/ 
specification definition, development, maintenance, and 
operation) . 

Table C-4. fchedul 

e Guideline 

Schedule 

Characterization 

Effort 
Factor (f^) 

Fast 

1.15 

Optimum 

1.00 

Slow 

0.85 


TO illustrate, assume that the development organization must 
develop a system estimated at 25,000 executable LOC. It is 
similar to ones they have developed before (old project 
type) , but it is being developed for a new computing fa- 
cility (new environment type); therefore, fj^ => 0.65. 
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Fifty percent of the code can be reused without modification 
(N » 0.5). The development organization has a team in mind 
whoso weighted applicable experience is 6 years (f 2 " 0.8) 
and the luxury of a slow schedule (f 2 “ 0.85). Then* as- 
suming that all other factors are normal (fj^ « 1, i “ 4 to 
k), 


= 8.0 (0.65) (0.8) (0.85) 
* 60.7 staff-months 


{[ (0.8) (0.5) + 0.2] 25.0}^*°^ 

(C-3) 


Since it is the beginning of the project, there is an uncer- 
tainty limit of ±100 percent in the system size estimate. 
Therefore, 

To t 10^ 

E (upper limit) = 3.536 {15.0} ’ x (1.0 + 1.0) 

» 121.5 staff-months (C-4) 

(lower linut) « 3.536 {15.0}^*°^ / (1.0 + 1.0) 

« 30.4 staff-months (C-5) 

Table C-1 (page C-4) lists the effort uncertainty limits 
that can be expected at the end of each life cycle phase. 
Figure C-1 illustrates the effort uncertainty limits as a 
function of phase. It is not sufficient for managers to 
merely state uncertainty limits. They must be able to 
explain why these Limits can be reached, i.e., what factors 
are uncertain, how they can increase or decrease the 
estimate, and by how much. 
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Size and Effort 



ORIGJNAL page 55 
OF POOR Quality 



t.s 


CAltNOAR TIMf ^ 


Figure C-1. Effort Uncertainty Limits as a Function of 
Phase 
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Schedules 


ORIGIK’AL !S 

OF POOR QUAUT\^ 

C.2 ESTABLISHING REALISTIC SCHEDULES 

The literature contains many models for resource estimation 
that are simple to implement but take some time to under- 
stand and calibrate for a particular environment. Few, how- 
ever, explain in any detail how to establish schedules from 
the information produced. Therefore, this subsection is 
probably more important for the more junior-lev'el manager. 

To establish realistic schedules, the development managers 
must basically consider the effort required for each ac- 
tivity, the team size, and the experience of the team leader. 

C.2.1 REQUIRED EFFORT FOR LIFE CYCLE PHASES 

In this example, the development organization computes the 
fraction of effort required for the design, coding, and 
testing activities from Equations (C-6) through (C-8) , re- 
spectively. 


fDAct _ 0.36N +0.04 
fCAct ^ 0.08N + 0.02 

= 0.36N + 0.14 


(C-6) 

(C-7) 

(C-8) 


where N is the fraction of newly developed and extensively 
modified code in the system size estimate and DAct, CAct, 

I 

and TAct are abbreviations for the design, coding, and test- 
ing activities. 

The development organization computes the effort required 
for the design, coding, and testing activities from Equa- 
tions (C-9) through (C-12) , respectively. 
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f = f^Act 

_j_ jCAct ^ ^TAct 

(C-9) 

gDAct _ 

(fD^ct/jj gTot 

(C-IO) 

pCAct _ 
E = 

(fC^ct^fj gTot 

(C-11) 

_TAct 

E ~ 

(fT^ct^^^j E?Ot 

(C-12) 


The development organization computes the effort required 
for each life cycle phase from Equations (C-13) through 
(C-18) , respectively. 


gRAPh ^0.15 (C-13) 

= 0.20 

= 0.40 

E^^^ =0.19 +0.94 + 0.57 (C-16) 

gSTPh _ gDAct ^ gCAct q^ 23 gTAct (C-17) 

gATPh _ gDAcc ^ ^CAct ^ gTAct (C-18) 


where RAPh, PDPh, DDPh, IPh, STPh, and ATPh are abbrevia- 
tions for the requirements analysis^ preliminary design, 
detailed design, implementation, system testing, and accept- 
ance testing phases, respectively. 
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Equations (C-i3) through (C-18) indicate that 75 percent of 
the design activity effort is expended to get to CDH;. nearly 
20 percent of the design activity effort is expended dur:ng 
implementation to respond to functional specification, de- 
sign, and implementation errors; and smaller ar, ounts of the 
design activity effort are expended during system and ac 
ceptance testing for the same reasons. The equations also 
show that nearly 60 percent of the testing activity effort 
is expended during implementation, 33 percent during system 
testing, and 10 percent during acceptance testing. 

C.2.2 REALISTIC SCHEDULES 

Once the effort for each phase is known, the development 
manager must determine how fast th development team can be 
staffed, how large it can get, and how fast team members can 
be released without losing control of discipline and order. 
This determination has to be made based on the project 
leader's experience level. In this example. Table C-5 pro- 
vides the development organization with a simple guideline 
for determining team size in terms of the experience of the 
team leaders. Table C-6 provides a simple guideline fcr 
producing a staffing pattern. 

Using these guidelines, the manager will find that team size 
peaks at the beginning of implementation. When the team is 
too large for a less senior project leader, the development 
manager can replace the project leader with a more senior 
project leader or extend the schedule. When the team size 
is too large for a senior project leade", the manager must 
extend the schedule or partition the development effort int") 
several smaller projects. Forming smaller projects, of 
course, will present the manager with software integration 
problems and additional management and support charges be- 
cause each smaller project will have a project leader and 
its own reporting and support requirements. 
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Table C-5. Team Size Guideline 


Minimum Years of Experience^' 


Project Manager 


Project Leader 


App. Orq. Leader App» Orq. 


Leader 


Maximum 
Team Size 
Excluding 
Team Leaders 


6 6 5 

7 5 3 

6 4 2 


6 4 * . ’ 3 

5 3 1 

4 2 0 


7 ±2 

4.5 ±1.5 
2 ±1 


App. = Applicable experience, i.e., requirements/ 

specif ication definition^ development, maintenance, 
and operation. 

Org. = Experience with organization. 

Leader = Experience as project leader or manager. 


I 
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Tabli' C-(». St.affitui I'.u l.L'ri\ Guidolino 


I PROJECT lEAOER 

SCHEDULE 

TYPE 

DEVELOPMENT TEAM MEMBERS 

TYPE 

LEAD 

TIME 

(VVfENSP 

PHASE IN 
INCREMENT 
iWFEKSl'’ 

PHASE OUT 
INCREMENT 
IWEEASI*-’ 

MINIMUM LENGTH 
OF PARTICIPATION 
1 WEEKS! 

SENIOR 

< 

FAST . 

\ 

•s 

6 


6 

OPTIMUM 


3 

6 


e 

SLOW 

3 

4 

6 

INTERMEPIAtE 

6 

FAST 

2 

3 

7 


d 

OPTIMUM 

3 

, 4 

7 


7 

SLOW 

4 

' ; s 

T 

7 

JUNIOR 

«( 

FAST 

3 

: 4 

8 


7 

OPTIMUM 

4 

5 

8 


S 

SLOW 


fl 

8 


"imu 1H.\T IHl. rnO.llOT MANAO.IH and UADIU NFtO TO OHC,ANi;:i: AND I’LAN PHOJFCIS lUFORE OrUER 
Tf AM MIMULHS JOIIM THl I'HO.ItCt 

‘’fasti Sr UAU AT WMIv'M TEAM Ml'MlUHS AHE AlTOtD TO TUT PWOJU'T SO THAT THE PIIO.IFCT LEAPER 
CAN MAINTAIN OIUHR 

‘ TASirSl ITAU AT VVMtCM U AM MFMSIHS lEAVE THE PtUMU'T SO THAT THE RETAILS OT THEIR 
ASSIliNMfNIS iVl HE AdSOHHfO (IT THE TEAM ANO M'NIMI/[ CAIIHACK. 

















Resources 


C.3 ALLOCATING PROPER RESOURCES 

Junior first-line development managers do not usually have 
much influence in forming the development team. A higher 
level manager in the development organization generally se- 
lects personnel based on his/her judgment and experience 
with the problem and discussions with the first-line man- 
ager. The first-line manager # however, must become familiar 
with the judgments made and the experience base available, 
so that he/she can organise the project effectively. 

In this example. Table C-7 provides the development organi- 
zation with a simple guideline for determining the type and 
experience level of personnel needed in terms of the com- 
plexity of the problem. Table C-8 provides a simple 
guideline for allocating resources in terms. of manager, de- 
veloper, and other support charges. 





Resources 




Table C-7. Development Team Staffing Guideline 


Project Environment 

Type^ Type^ 


Percentage of Percentage 

Senior Personnel” of Analysts 


Old 

Old 

25-33 

25-33 

Old 

Nev; 

33-50 

25-33 

New 

Old 

33-50 

33-50 

New 

New 

50-67 

33-50 


^The project and environment types are old when the develop- 
ment team has more than 2 years of experience with them. 

K 

Senior personnel are those with more than 5 years of ex- 
perience in development-related activities. 


‘^Analysts are those personnel who have training and an educa- 
tional background in problem definition and solution with 
the application (project type) or the computers (environment 
type) depending on the problem. 
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Table C-8. 


Development Staff Composition Guideline^ 
(l,ofv2) 


I ; 

i 


Personnel Percentage 

Category of Staff Funct ion 


MANAGERS 

Project 

Manager 


Project 

Leader 


Administrative 


Customer 

Interface 


•21.5 

5.0 First-line development 

manager responsible, with 
project leader, for organi- 
zation and planning of proj- 
ect. Provides technical 
consultation and manages 
project resources. 

7.5 Lead developer responsible, 
with project manager, for 
organization and planning of 
project. Provides technical 

' direction and day-to-day 
supervision of project ac- 
tivities. 

4.5 Second-, third-, fourth- 
line development organiza- 
tion (higher level) managers 
and project control office 
personnel responsible for 
administrative aspects of 
projects, such as financial 
reporting and quality assur- 
ance of documentation and 
progress report formats and 
procedures. 

4.5 • First- or second-line manager 

from customer's organization 
responsible for monitoring 
resources and progress. Is 
primary point of contact 
vdth external customer, sup- 
port, and contractor groups. 


f : 

> • 

^ i 


^This breakdown is for development charges. Other charges to 
the overall cost of a project come from staffs with analo- 
gous breakdowns, e.g., the requirements team, the independ- 
ent verification and validation team, the maintenance and 
ooeration team. 
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Table C-8. Development Staff Composition Guideline^ 
(2 of 2) 


Personnel Porcontac?e 

Category of St a f i: 


Funct ion 


DEVF.LOPF.RS 


63.5 Programmer/analysts respon- 

sible for requirements anal- 
ysis, design, implementation, 
testing, and documentation 
of software. 


SUPPORT 15.0 

Librarian 5.5 

Secretarial 4.0 

Publ i.-?at ions 5.5 


Personnel responsible for 
development-related cler ica 1 
work and source code mainte- 
nance . 

Personnel responsible for 
development-related and de- 
velopment organization 
office clerical work. 

Reproduction , composition , 
graphics, and editorial per- 
sonnel responsible for 
formal production of docu- 
ments and reports. 


This breakdown is for dove lv'<pment charges. Other cliarges to 
the overall cost of a project come from staffs with analo- 
gous breakdowns, e.g., the requirements team, the independ- 
ent verification and validation team, the maintenance and 
operation team. 
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Organization Summary 
C.4 SUMMARY 

Although the example in this appendix is not a recommended 
or even a suggested procedure for organizing aspects of a 
project, it provides the junior-level development manager 
with basic information about organization that is not avail- 
able v;ith simple estimation models. The type of information 
emphasizes the need for managers to record development in- 
formation and judgments through the development plan, data 
collection, and postmortem evaluations. 

Managers must realize that guidelines are just that and must 
not let their education obstruct their common sense. 

Blindly applying guidelines to solve a particular problem 
frequently leads to undesirable situations. Guidelines 
should only provide a starting point from which a manager 
makes common sense adjustments to suit project-specific con- 
ditions. 
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PnOCESS PHASES 


REQUinEMEMTS 

ANALYSIS 


PRELIMINARY DETAILED 
DESIGN DESIGN 



SYSTEM 
INTEGRATION 
AND TESTING 


ACCEPTANCE MAINTENANCE 
TESTIN G AN D OPEH ATI ON 
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Ni>T€ FOR eXAMPlP AT TH« END OF THE IMPLEMENTATION PHASE UTM OASMEOIINEI. APPBOXIMATCLY T9S OF TM| STAFF ARE INVOLVED IN 
SYSTEM iNUr.RATlON AND TEST'NG APPROVMATEIY ARf ADDRESSING nEOVIRMENTS CHANGES OR PROBLEMS. APPROXIMAT El> 

ABE designing MODIMCATICNS and APPHOXIMATELV its are coding and unit testing changes 
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DEVSLOPrt^SEyT 


ACTiOr^S AS^D TS^Af^SACTtOWS 


ACT5VSTIES J5of«waro Dovo5cpi«cn« PIcn Prcpnrod 

FunsHons! flcqulrcmonta AmpUfJod 
Pcrfonr.enco tlc^ijulrsmsRfcJ AstaSyassd 
OporatiomsI Retjisiremonea OstGTvnincd 
Intcrfacas ldantJR®d 

Report end Oleptay Ro^^uSrementa DoCertnlnsd 
TOD Rcejulrsmants Idontift&d 
Oystsm SI 20 EstSmotsd 
ReusoSiFj Softi«>sre §dsr.tJflcd 
Cemputof Resources Dotsrminod 
t'Scrtfwcio Coteotad 

Roquiromanta AinRlyeia Summary Report Frsparod 
SRR Hsid 

OWD PRODUCTS? Softwero Dawelopmant Pten 

Repuiromortta Anaiysla Summer/ Report 

R^ETMODOLOGtSS Pro}oct Kotsbook 

Data CoSiection ; 

UbrcTisne 

Unit Dsvoiopmont Folders 

Requlrementn Quoatton end Chanda Racorda 

Structured Ar.alyaio (CompicK Data Procecaktg) 

TOOL® Conflgurntion Ancivola Tool (CAT) 


» 103 IS 1>-£3 
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R^AEWAGEEVSERIT 

Adsorbs AWD TKAFJSACTEOiyS 

ACTJVmES 

Eofewaro DoveJopmnnt Ptan Roifiowari 
Schsduio end EtaCHng PSsnnod 
RequiroRtonts Analysis Summary ReporS Rcvlowed 
Prctimincry Dass^jn Trr.ns5t5en Planned 

Team Trotnod 

SteRdsrda and Pmccduras Hnf oread 
ProQross P.'lonitorcd 
Visibility Promoted 
Syatem Size Estimated 
ncscurcce and Cost EstSmatod 
Team Interaction Coordinated 

aiEAsuriES 

TBD Requlromants 

Raquiroments Questions end Answers 
Requiromonts Chenoos 
Subjactlvo Evaluations 
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ACTeoriis Ar^D TSArjSACTiorjs 


Syatfflsn PartJtJoncd 
Proccocsng Opsone Defined 
AltsmntJvo 0?45ssns EKRminod 
E^emsS 5n«orfa-30s DafJnad 
Subsystems Pertitionod 
Subsystem Cntorfscsa Ssfined 
Enrer Processing end Rscovory Defined 
TDD Rsijitirenisnta Rcsoivsd 
ncussbie Softivcro tdsntin&d 
Profiminary Dsdsn Report Prepared 
PDR HsJd 


Preiiminar/ Design Report 
Software DovcSopmsnt P2an Update 


Prcjcjct Notebook 
Data Coiicetion 
Llbrariaris 

Unit Development Foidara 

Requirements Question and Chango Records 

Design FormsIiJtms 

Design Decision end Chengs Records 

Configuration Wenagsmont 

Design Wal!:throughs 

Itorctivo Enhancement 

Infcrmsticn Hiding 

Data Abstraction 

PDL 


PDL Processor 

Source Code Library Management System 
CAT 

Rosourco Estimation 
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Asrrsorsis aku trawsactiows 

ACTJVtTiES 

TQD netjuircmstuts Rcsotvod 
no^uiromsnts Chc.ng{>s Rsviowed nnd Acsossod 
Dffsign Roviawod end Wclkcd Through 
Dotallod DosSgn Transition Fisnned 

Teem Treinsd 

Ctandordo end Procedures Enforced 
Progroco P/ionitorod 
Vicibiitty Promotad 
Syntom S!se Estimated 
Rssourccs end Coot Estimated 
Team interaction Coordinated 

EViEASURS-S 

TOD nequiroments 
RequiremontK Changes 
RoquSrcments Questions end Answers 
Dosign Chongss 
Interfecas 

Design Completion Checklist 
Subjoctiva Evaluations i 
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DEVELOPSVJEWT 

ACTSOl^’S AMD TOArjSACTIO£\!S 

ACTiViTlEC 

SIngCa Functions Refined 
Oacstinc Diegrams Prepared 
I/O Spscified 

PDL end Prologs Spscifiod 
COMMON Blocks Spocifiod 
Interne! Intoi'fssso Specified 
TSD Roquiroments Recolvcd 
Rcii&ebla Softisrans Sdcntined 
Dotnilcd Design Document Propsrad 
Impiomsntetion Strategy PIsnnod 
CDR Hold 

END PRODUCTS 

Detnilod Design Document 
Software Dovolopment Plan Updata 
ImpJomsntation Plan 

METHODOLOGIES 

Project Notobook 
Data Colioctlon , 

Librarians 

Unit Development Foldors 

Roquirements Question end Change Rocords 

Dasign Formalisms 

Design Decision and Change Records 

Configuration IVIonagomont 

Design VValkthrougha 

Iterative Enhancement 

Informntion H’ding 

Deta Abstraction 

PDL 

TOOLS 

PDL Processor 

Source Code Library Management System 
CAT 

Resource Estimation 
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IVJAWAGEiVilEt^T 

ACTIONS AWD TRAWSACTIOI^S 

ACTIVITIES 

Impicm&ntcticn Strotonv Rovinwad 
Implamontatlon Transition Plannad 


TOO itemo Rotsoltrad 

Roqulroments Changss noviowed end Aesossed 
Oosign Reviewed end Welkcd Through 


Taem Trained 

Standards end Prccoduroo Enforced 
Progreea Monitored 
VicIblUtv Promoted 
System Sixo Eetimntod 
nosourcGS and Cost Estl.'nntod 
Toam Interaction Coordinntod 

WJEASUFIES . 

TGD Itcme 

Reqiiiromenta Changes • 
Requirements Questions and Answers 
Design Changes 
IntsrfacoB ; 

Design Completion Checklist 
Design Growth P.ats 
Modulo Strength 
Modulo Coupling 
Subjactivo Evaluaticno 
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DEVEL0PW3EJ\!T 


ACTE0R3S ARED TRAEySACTJOWS 


ACTIVITIES Job Confthc! Uinguaea Prepared 

Command Proceduros Prepared 
I i^ow H^odubs Coded 

Rouaoblo Moduloa Rovlasd 
Units Integrated end Tasted 
Suild/neloesa Tost Ptana Prepared 
Data Prepared 

□uild/Rstoaco Test Pians Enscutsd 
DIscropencloo Ronolved 
System Integrated 
System Tent Plan Prsparod 
Acceptance Teat Plan Prepared 
Ueer'e Guido Property 
System Description Prepared 

END PRODUCTS System Code 

Supporting Data and System Flies 
DuEtd/Releass Test Plana end Raaults 
System Tost Plan 
Acccptanco Test Plan 
Droft User's Guido 
Droft System Doocriptlon 
Softwaro Dovolopment Plan Update 

METHODOLOGIES Project Notebook 

Data Collection 
Librarians 

Unit Dovelopmont Folders 
Requiromenta Quostion and Change Records 
Design Dacision end Change Records 
Coding Standards 
Structured Code 
Code Reeding 
Code Change Records 
Configuration Management 
Builds/Reloaooa 
Top-Down Implementation 
Formal Test Pinna 
Functional (Thread) Testing 

TOOLS POL Processor 

Source Code Library Mansgomont System 
Structured Coding Language 
CAT 

Resource Estimation 
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mAWAGER/JEi^T 

ACTEOraS AIMD Tr?AI\!SACTEOWS 

ACTIVITIES 

CuHd/ncteaso Test Plans Rovlavyod 
Culld/Rcloaito Toot Plan Results Rovlowcd 
Discrepancies Bosolved 
System Test Plan Revlowed 
Draft User’s Guido Rovlowcd 
Draft System Draocription Rovlowcd 
Systam Tooting Transition Planned 

TGD Itsms Resolved 

Requirements Changes Roviowod and Assessed 
Design Changes Rovlowcd and Vf^alkod Through 
Cedo Chengos Rovlowcd 

Taom Trained 

Stenderds and Procedures Enforced 
Pregross Monitored 
VlclbiJItv Prometsd 
System Size Estimated 
Resources end Cost Estimated 
Toam Interaction Coordinstod 

MEASURES 

TGD items 

Requirements Ch’tngos 
Roquiromcnls Questions and Answers 
Design Chengos 
Coda Changes 

Code/Tost Completion Checklists 
Code Growth Roto 
Error/Chengo Growth Rates 
Oiscropancics/RosoEutions Growth Rotas 
Computer Usage Growth Rate 
Toam/Individual Productivity Rates 
Subjoctivo Evaluations 
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DE¥EL0E*!V3E[yT 

ACTBOE\!S AMD TRAR3SACT10S\SS 

ACTJVITIES 

System Croatud 

System Tost PSon EKOcutcd 

Discropcncios ncsoSvod 

Uccr's Guido Roviowod end Rovisad 

System Dsscriptlon Reviewed and Rovised 

Acccptcneo Tostinn Plonned 

mD PRODUCTS 

System Codo Updots 

Supporting Data and Syetom riles Update 

System Tost PIr.n Results 

User's Guido Update 

Systom Doscription Update 

Softwsro Devolopmont Plan Update 

rWETHODOLOGIES 

Project Motobook 
Data Collection 
Librarlons 

Unit Dovotopmont Folders 

Requiremonts Quostion and Chango Rocords 

Design Decision and Change Records 

Codo Chango Rocords 

Configuration Management 

Formal Tost Plan 

Functional (Thread) Testing 

TOOLS 

PDL Processor 

Source Codo Library Managoment System 

Structured Coding Languego 

CAT 

Roaourco Estimation 
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aflAI\3AGEtV3EWT 


ACTIONS AHD TRAP^ISACTBOr^JS 


ACTIVSTIES Syctom Tost Pfjtn Rosults Rovlawcd 

Dtacrcpcncios Resoiuod 
Acceptcnce Toot Pian Haviowod 
Accoptonca Tooting Transition Plonnod 

User's Guide Rovlowod 
System Dsscription Rovlowod 

TBD Items Roscivcd 

Requirements Chenges Rovlewod and Assossod 
Dseion Chongos Raviowed and Wsikod Through 
Code ChengoB Rovlowod 

Team Trained 

Standards end Procoduroo Enforced 
Progroas Monitored 
Visibility Promoted 
System Sizo Estimatod 
Resources and Cost Estimatod 
Tosm Interaction Coordinated 

MEASURES TBD Items 

Roqiilrements Changes 

Requiromonts Quostlons and Answers 

Design Changos 

Coda Changes 

Tost Complotion Chockllst 

Coda Growth Rata 

Error/Chango Growth Rates 

Discropancios/Rosolutiona Growth Ratss 

Cemputar Usage Growth Rata 

Taam/Individual Productivity Rotas 

Subjoctivo Evaluations 
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DEVELOPIViEWT 

ACT!0?JS AWs5 TRAr^'SACTSOWS 

ACTIVITiES 

Systom Crontod 

Ucerp ond OjJorators Treinod 

Acceptcnco TccC Plon Exocutad 

Oiscrspcnsies Rocaived 

Uaar's Giiida Reviewed and Rovicad 

System Dsscriptisn Raviswsd ond Rovlsad 

Systom Dolivory Plannod 

ORR Held 

Sottwero Dayclepmont History Prepared 

END PRODUCTS 

Syatam Cods 

Supporting Data end Syctsm Files 
Accoptonco Tust Plan Results 
User's Guida . 

System Description 

Archived Systom (Topos) end Documentation 
Softwaro D&volcprnant History 

METHODOLOGIES 

Project NotobooU 
Datn Collection 
Librarians 

Unit Dcvolopmant Fcldors 

Requiremonts Question and Chengo Records 

Design Docision and Chango Rocords 

Coda Chenge Rocords 

Configuration Managsment 

Formal Tost Plan 

Functional (Thread) Testing 

TOOLS 

PDL Processor 

Source Code Library Mnnagomont System 

Structured Coding Language 

CAT 

Rosourco Estimation 
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IV3ArJAGEr,/3EI\ST 


ACTSOWG AWD TfSAfySACTIOWS 


ACTiVITiES Accsptonco Test Plan Results Rovlowed 

DIscropcncics Received 
Syetom Dativory Reviswod 

User's Guido Roviovtf&d 
System Doscriptlon Reviowod 

TOD Itoms Resolved 

Rstsuiromonts Chnnocs Reviewed und Assossod 
Dosign Changes Roviewod end Walked Through 
Code Chsngaa Reviewed 

Team Trained 

Standards and Procedures Enforced 
Progrocs F^onitorod 
Visibility Promoted 
System Siso Estimated 
Resources end Cost Estimated 
Team Interaction Coordinated 

MEASURES TOD Items 

Requirements Changes i 
Raquiromants Qusstlons and Answers 
Design Changes 
Code Changes 
Tost Completion t'hochlist 
Codo Growth Rato 
Error/Chango Growth Rotes 
Oiscropsneios/Rosolutions Growth Rates 
Computer Usoge Growth Rato 
Toam/Individual Productivity Rates 
Subjoctivo Evaluations 
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o Fpequisrscy of SGliedulG/iVSIJostoiis Chnrtges 

o ConsEstencv in Organisatlona! Strwcti:?© Compared 
With Origisiai Pians 

© RuotMatioa in Project Sta«^ Level and System Size 
Estimates 

o ■ E*Sistory O'? i^umber and Type of TBD items for 
Requirements and Design 

o Ease of Access to information on Project Status, 
Schedules, and Plans 

o Frequency and Amount of iir.usualiy Long E*lours 
Required or Planned To Attain Certain Objectives 

© Level of Detail (Both Technical and fi/ianageria!) 
Ifnderotocd and ControHod by the Project R/ianager 
and the Project Leader 

& Discrepancies in Staff Level and Workload 

© Discrepancies in Planned Weekly Steff Level and 
Computer Usage or Compared With Past Projects 




O ScheduBed CspaBsliltlos OsBayed to Lotor B&fsBd/Reioese 

o Coding Started Too Early {Staff Too Largo Too Early) 

o I^Sumorous C?iangos Mod© to Enstlal Software' 
DovolopjTtsnt Plan 

o Guidelines or FBannod Froo©dyros Oeomphaslzcd or ' 
Delotod 

o Sudden Changos In Staffing (Magnitude) Suggested 
or Made 

o EKcessIve Oocumontation and PaperworEc That Mavo 
Little Direct Oearlng on Eesiuired Documentation 
Prepared ■ ■ - 

© Continual Bncreaso in l^umbsrs of TBD Items and 
ECBs Measured 

© Decrease In Estimated Effort for System Testing 
Suggested or Made 

© Reliance on Other Sources for Soon-To-Be-Availabla 
Software 









o Stop CtarrasiU: ActsvitiaQ and Rovfaw/CompiQto 
PredscQssor or Problom Activity 

o Docroaso Staff to R^anageabSe Lovel 

o flcpiaco Jursioir With Senior Pornonrss! 

o IncrGose and Tightsn iVianagement Procoduires 

© Encroaso J^umhor of intormodtato DQliverablos 

© Decresss Scope of Work and Define a EVianagaablo, 
Doable Thread of tho System 

o Audit Project with Endopondent Personnel and Act 
on Their Rndings 






' © Us© a SmaSJ Senaor Stairf for tSie EsrEy Lif© Cycle 
Phases 

O Develop and Adhor© to a Software Development Plan 

O Define Specific intermediate and End Frodiscts 

o Enamine Aiternative Approaches 

© Use Formal Testing 

o Use a CentraS Repository 

© ICcep a Detailed List of TBD items 

o Update System Size, Required Effort, Cost, and 
Schedule Estimates 

0 Allocate 30 Percent of Effort for Integration and 
Testing 

o EKperiment 
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o Don't Ov©r 25 taff 

o Don't AHow Team iVSembers To Proceed in an 
Undisciplined iVianner 

o Don't Deiegate Technical Details to Team iUlembers 

o Don't Assym© That a Rigid Set of Standards Ensures 
Success 

o Don't Assume That a Large Amount of Documenta- 
tion Ensures Success 

o Don't Deviate from the Approved Design 

o Don't Assume That Relating Standards Reduces 
Costs 

o Don't Assume That the Pace WiW Increess Later in 
the Project 

o Don't Assume That intermediate Schoduis Slippage 
Can Be Absorbed in a Later Phase 

o Don't Assume That Everything Will Rt Together 
Smoothly at the End 
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o a Fv/rat£3ri Software DeveSopmsnt Flan. Being 
FoSSowed? 

O Are Life CycEe Phases and Prodticts Defined? 

o js Someone In Charge? 

o Docs the Staff Sizo IViatch the Workload? 

© Do Team R/lembsrs Know Where the Project Is and 
Where It Is Going? 

o Is a Conflguraticn Control Plan Being Followed? 

o is There a Single Complete List of TBD Stems With 
Assessments? » 

I 

© is There a Commonly-Adhered-To IViethodology? 

o E-3ave Alternative Designs and Approaches Been 
Ccnsideroci? 

© Are There Contingency Plans for Rationally Solving 
Problems? 
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SRR PFSESEWTATiORl RSATEBSAL 


O Introduction and Agenda 
o Roquiremonts Summary 
O Analysis Ovorviow 
o Functional Specifications 

— Environmontai Considsrntions 
— Oparational Roquiromants 

(1) Operating Scenarios 

(2) Data Row Analysis 

(3) PsrformancQ Requirements 

(4) Interface Roquiromonts 
— Requirements Relationships 

o Derived System Requirements 
o Requrroments Pt^anagement Plan 
O Personnel Organization and Interfaces 
© Software Performance and Testing Requirements 
o Issues, TBD Items, and Problems 
O IVlilestonos and Suggested Development Schedule 


SRR FORR/IAT 

PRESEWTERS 

Requirements Definition Team 

PARTICIPAWTS 

Development Team Representatives 
Quality Assurance Representatives 
User Representatives 
Customer Representatives 

TIME 

After Functional Specifications Completed and 
Before Functional Design Started 

HARDCOPY 

DISTRIBUTION 

Minimum of 5 Days Before SRR 
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1. Introduction 

2. Roquiromonts Suinninry 

3. Anniysis Ovorviow 

4. Functionnl Sfjocificatloiis 

a. EnvironmontnS Considorations 

b. OperntionnI Roquiromonta 

(1) Oporoting Sconcrios 

(2) Dntn Flow Analysis 

(a) System Input 

(b) Procossing Roquiromonts 

(c) System Output 

(3) Porformnneo Roquiromonts 

(4) Intorfneo Roquiromoiits 

c. Roquiromonts Relationships ; 

5. Derived System Roquiromonts ' 

6. Utility, Support, and Tost Programs I 

7. Reusabio Software Summary 

8. Data Set Dofinitions 

9. Roquiromonts Mnnagomont Plan 

a. Personnel Assignments 

b. Description of Required Documents 

c. Configuration Control Approach 

d. Enhancomont/fVlaintonanca Procodures 

o. Reporting and Testing Evaluation Procedures 

10. Personnel Organiration and Interfaces 

11. Software Performance and Testing Roquiromonts 

a. Analytical 

b. System 
n. interface 

d. Acceptance 

12. Issues, TBD Items, and Problems 

13. fViilestonos and Suggested Development Schedule 
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^ Introduction and Agsnda 
o Dosign Ovorviovv 

0 High-Lovol Disgroms of Operating Scenarios 
O High-Lavol Diagrams of Syotom Structure 
o IVInjor Softwara Components 

— Kigh-Lovoi Diagroms of Subsystems 
— High-Lovol I/O Spocifications and Interfaces 
— Functional Basolino Diagrams (Troecharts) 
o Design Team Assessment 

O System Size, Required Effort, Cost, and Schedule Estimates 
o ResourcG Allocation and External Support 
o Development EVlanagoment Plan 
o Personnel Organization and Interfaces 
o Testing Strategy 
o l&suos, TBD Items, and Problems 
o IVeilostonos and Schedules 


PDR FORfVlAT 

PRESEWTERS 

Software Development Team 

PARTICIPAWTS 

Requiromonts Definition Team 

Quality Assurance Roprosentativos From Both 
Groups 

Customer Interfaces for Both Groups 

TIME 

After Functional Dosign Completed and Before 
Ootailod Dosign Started 

HARDCOPY 

DISTRIBUTION 

Minimum of 5 Days Before PDR 
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PDR HARDCOPY R'iATERBAL 


1. Introduction 

2. Design Overview 

3. High-Level Diegrems of Operating Scenarios 

4. High-Level Diagrams of System Structure 

5. Critiqua of Alternative Designs 

6. iVlajor Software Components 

a. High-Level Diagrams of Subsystems 

b. High-Level I/O Specifications and Interfaces 

c. Functional Baseline Diagrams (Troecharts) 

d. Screen, Printer, and Plotter Formats 

7. Hardware Interfaces 

8. Internal Data Set Definitions 

9. Reusable Code Summary 

10. Design Team Assessment 

11. System Size, Required Effort, Cost, and Schedule Estimates 

12. Resource Allocation and Externa! Support 

13. Development IVIanagemont Plan 

a. Life Cycle and Products 

b. IVlQthodologios 

c. Models and Tools 

d. Configuration Control Approach 

14. Personnel Organization and Interfaces 

15. Testing Strategy 

a. General Approach 

b. Extent 

c. Control Mechanisms 

16. Issues, TBD items, and Problems 

17. Milestones and Schedules 
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O Entroductlors STid Agenda * 

■ o Design OvopvSew 

o E-ligh-LovaH DlEgremc of Operating ScGnnrios 
o High-LoueJ Diagrcmc of System Structure 
o EVSajor SofJwara Components 

— Efigh-LoveS Diagrems of Subsystems 
— Kigh-Lovo! I/O Specifications end Interfaces 
— Functional Basoiino Disgremo {Treoebarts) 

— Error Procoasing and necovery Strategy 
— Restrictions of Procossing Pv'iodQS 
— internal Storags Requirements 
© Design Team Areossment 
© ImpSomontation Strategy and Traoeabslity 
o System Sise, Required Effort, Cost, and Schedule Estimates 
o RGSOurco Allocation and EKtema! Support 
o Development R^enagomont Plan ; 

© Portonno! Orgeniaotion and Interfaces 
o Testing Strategy 
© Issues, 7BD Iterns, end Problems 
o [V]i!estoncs and Schedules 



PRESENTERS 

Software Development Team 

PARTICEPAWTS 

Requirements Definition Team 

Quanty Assurance Representatives From Both 

Groups 

Customer Interfaces for Both Groups 

TIME 

After Detailed Design Completed and Before 
Implementation Started 

HARDCOPY 

Minimum of 5 Days Before CDR 

DISTRIBUTION 
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1. introduction 

2. Design Ovorviow 

3. High-Level Diagrams of Operating Scanarios 

4. High-Level Diagrams of System Structure 

5. R/iaJor Software Components 

a. High-Level Diagrams of Subsystems 

b. High-Level I/O Specifications and Interfaces 

c. Functional Baseline Diagrams (Trocchorts) 

d. Error Processing and Recovery Strategy 
o. Restrictions of Processing iViodcs 

f. Internal Storage Requirements 

g. Detailed I/O Specifications 

(1) Processing Control Parameters 

(2) Screen, Printer, and Plotter Formats 

6. Hardware Interfaces 

7. Internal Data Set Definitions 

8. Reusable Code Summary 

9. Design Team Assessment 

10. impiementation Strategy and Traceability 

11. System Size, Required Effort, Cost, and Schedule Estimates 

12. Resource Allocation and External Support 

13. Development iVlanagement Plan 

a. Life Cycle and Products 

b. Methodologies 

c. Models and Tools 

d. Configuration Control Approach 

14. Personnel Organization and Interfaces 

15. Testing Strategy 

a. General Approach 

b. Extent 

c. Control Mechanisms 

16. Issues, TBD Items, and Problems 

17. Milestones and Schedules 
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RE¥0EW 




'Or^E: 




SER)T 


o Introduction ond Agendo 
o Systom Roquironiohts Summnry 
O Annlysio Ovorviaw 
o Support System Ovorviow 

— IVlcjor Softwnro Components 
— Systom Testing Philosophy 

— System Testing and Porformanco Evaluation Results 
— Roquiromonts Verification Philosophy 
— System Softwaro and Documontation Status 
o Operations and Support Plan 

— Porsonnol Assignmonts and Rosponsibilitias 
— Organisational Intorfacos 
— Data Availability 
— Facilities 

— Operating Scenarios 
© Systom hlanagomont Plan 
© Porsonnol Organization and Intorfacos 
O Issues, TBD Items, and Problems 
o Contingency Plans 
s> Miiostonos and Timclino of Evonts 
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ORR HARDCOPY EVJATERSAL 


1. Introduction 

2. System Roqusrcmants Summary 

3. Analysis Overview 

4. Support Systonj Overview 

a. fWajor Softi<varG Compononts 

b. System Tooting Philosophy 

c. System Testing and Porformanco Evatuation Results 

d. Requirements Verification Philosophy 

o. System Software and Documentation Status 

5. Oporations and Support Flan 

a. ForsonnsI Assignments and Responsibilities 

b. Organizational Interfaces 

c. Data Availabiiity 

d. Facilities 

ID formal Oporations 

(2) Critical Operations I 

13) Emergency Ope' ’ons i 

(4) Contingency Operations 
G. Operating Scenarios 

(D Support Requirements 

(2) Timeline of events 

(3) Operating Procedures 

(4) Resources Required 

6. System Management Plan 

a. Personnel Assignments 

b. Description of Required Products 

c. Configuration Control Approach 

d. Enhanccment/Maintenanco Procedures 

e. Reporting and Testing Evaluation Procedures 

f. System Performance Evaluation Procedures 

7. Personnel Organization and Interfaces 

8. Issues, TBD Items, and Problems 

9. Contingency Plans 

10. Milestones^and Timeline of Events 
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PROJECT ^^OTSBOOEC mBWiAJ 

1. Descs'iption and Last of IVlajor Components 

2. ECey PersonneS and Their Responsihilitios 

3. Description of Capabilities in Each Build /Rcicase 
Deliverehlo Products 

5. History of Events, Schedules, and IVlilestones 

6. E*Sistory of System Size, Required Effort, and Cost 
Estimates 

7. History of Source Code Changes 

8. History of Accomplishments 

9. History of Outstanding TBD Items, Changes, Errors, 
and Problems 

10. History of Verification of System Requirements 
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4. Areas of Concern end TBD Requirements 

5. AnaSysis of Basic and Derived System Rcquiremonts 

6. AnaSysis of Basic and Derived Requirements by 

Subsystem < 


8. Summary of Rousabio Software 

9. System Size, Required Effort, Cost, and Schedule 
Estimates 








’3. isitroslucttors 

a. OvorsS! System Concepts 

b. I-Cigii-Low©? Fsctajras of System 

c. Operating Scenanos 

d. Design States 

e. Critsqno of Altomativo Designs 

2. Subsystems 

a. Hsgh-LsvsS Pictures 

b. Ooscrlptlon of input and Output 

c. Description of Processing 

d. Functions! Baseline Diagrams 
©. Prologs and PDL for First Love! 

3. Resource Requirements 

a. Hardware 

b. Data Definitions 

c. Peripheral Space Considerations 

d. Rflemory Considerations 

4. Data Interfaces 

a. Description 

b. Format 

5. Summary of Reusable Software 



















DETAILED DESlGf^ OOCOE^EI^T FOKSVIAT 

1. SntrodisctBon 

2. Subsystems - 

a. Overall Capability 

b. Kegb-Lavei Pictures 

c. Oescrlption of input and Output 

d. Description of Processing 

0. Baseline Diagrams ; 

f. Restrictions 

g. Internal Storage Requirements 

h. Datailed Input and Output Specifications 

1. List of f^iessagos 

j. Description of COIVilVIOEy Areas 

k. Prologs and PDL 

3. Resource Requirements 

4. Data Interfaces 

5. Summary of Reusable Software 

6. Results of System (Vlodeiing 
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TEST PLAM POEMKT 

1. Introduction 

a. Purpose 

b. Typs and Level of Testing 

c. Schedule 

2. For Each Test 

a. Purpose 

b. Detailed Specification of Input 

c. Required Environment 

d. Operational Procedure ; | 

e. Detailed Specification of Output ' 

f. Pass/Fai! Criteria 

g. Discussion of Results 
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1. isitroductlon 

a. Oworali System Concepts 

b. Hlgh-Lsve! Pictures 

c. Operating Scenarios 

2. Subsystems 

a. Overal! CapabiSety 

b. Assumptions and Restrictions 

c. High-Lovel Pictures 

d. Description of input and Output 
Q. Description of Processing 

3. Requirements for Hriecution 

a. Resources 

b. Run Information 

c. Control Parameter information 

4. Detailed Description of Input and Output 

a. Facsimiles of Graphic Displays 

b. Facsimiles of Hardcopy Output 

c. List of IViessages 
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SVSTE^^ DESCiHeraoai 

1. Irttroduction 

2. Subsystems 

a. Ovoraii CapabiSity 

b. Assumptions and Rostrictions 

c. High-Lovei Picturos 

d. Description of input and Output 
0 . Basolino Diagrams 

3. Requirements for Creation 

a. Resources , i 

b. Creation information 

c. Program Structure information ^ 

4. Detailed Description of Input and Output 

5. internal Storage Requirements 

6. Data Interfaces 

7. Description of COIV3IVIOi\] Areas 

8. Prologs and PDL 

9. List of Software from Support Libraries 
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SOFTOAKE OE¥ELOPa^EE^^T IhSiSTOBY 

FGRa/aAT 

1. ProjocS Dascriptsors and Backgroamd 

2. DavoSopmont MSs^ory 

3. Project Assessment 

4. FunctionnS Specifications and Roquiremonts 

5. Summary 
S. RoforoncQS 







Act 

App 

AT 

ATPh 

C 

CAct 

CAT 

CCB 

CUR 

CPU 

D 

□Act 

DD 

DDPh 

DEC 

E 

ECR 


f 

FSRD 

GESS 

GSFC 

I 

I AD 

IBM 

ICD 

Intel 

IPh 

I/O 

JCL 

K 


GLOSSARY 


activity ' ' "■ 

applicable 
acceptance testing 
acceptance testing phase 
coding 

coding activity 
Configuration Analysis Tool 
Configuration Control Board 
critical design review 
Central Processor Unit 
design 

design activity 
detailed design 
detailed design phase 
Digital Equipment Corporation 
effort 

Engineering change request, requirements 
modification request, functional specifications 
modification request, specifications modification 
reques t 

factor 

functional specifications and requirements document 

Graphic Executive Support System 

Goddard Space Flight Center 

implementation, code and unit testing 

interface agreement document 

International Business, Machines 

interface control document 

Intel computer 

implementation phase, code and unit testing phase 
input and output 
Job Control Language 
one thousand 


G-1 


L 

LOG 

M&DOD 

MEDL-R 

MO I 

MOU 

NASA 

NSP 

Org 

ORR 

PD 

PDL 

PDP 

PDPh 

PDR 

Ph 

POC 

PSA 

PSL 

RA 

RAPh 

RID. 

S 

SAP 

SEL 

SFORT 

SIRD 

SLOG 

SRR 

ST 

STPh 


T 


' ' ORIGINAL PAGE ?S 

OF POOR QUALITY 

adjusted lines of code 
lines of code 

Mission a lid Data Operations Directorate 

Multi-Level Expression Design Language - 
Requirements Level 

memorandum of information 

memorandum of understanding 

National Aeronautics and Space Administration 

NASA Support Plan 

organization 

operational readiness reviev; 
preliminary design 

Process Design Language, Program Design Language, 
pseudocode, metacode 

DEG computer 

preliminary design phase 

preliminary design review 

% 

phase 

point of contact I 

' 1 

Problem Statement Analyzer i 

Problem Statement Language j 

requirements analysis 
requirements analysis phase 

review item disposition, review item discrepancy 
system 

FORTRAN Static Source Gode Analyzer Program 

Software Engineering Laboratory 

Structured FORTRAN Preprocessor 

support instrumentation requirements document 

source lines of code 

system requirements review 

system testing, system integration and testing 

system testing phase, system integration and testing 
phase 

testing 
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TAct 

TBD 

Tot 

UDF 

VAX 


ORiGJNAL PAGE SS 
OF POOR QUALITY 

testing activity 

"to be deterniined" used as an adjective (TBD items) 
or as a noun^ (TBDs) 

total 

unit development folder 
DEC computer 
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